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PREFACE 


I  was  privileged  to  kick  off  the  Defense  Risk  and  Uncertainty 
Workshop  with  a  panel  that  speculated  on  how  well  DOD  analysts  and 
decision  makers  are  handling  risk  and  uncertainty.  The  problem  is 
of  course  quite  complex.  It  will  take  a  lot  of  good  people  and  a 
lot  of  good  ideas  to  improve  our  analysis  and  decision  making 
processes . 

A  forum  such  as  this  one  serves  to  surface  ideas  in  a  very 
efficient  manner.  You  will  note  in  these  proceedings  a  number  of 
different  approaches  in  handling  risk  and  uncertainty.  All  of 
these  papers  were  subject  to  th=  critique  of  peers  in  the  various 
panels  of  the  workshop,  and  I  feel  this  peer  group  interaction 
ultimately  provides  the  greatest  benefit  to  the  attendees. 

To  those  of  you  not  at  the  workshop  I  commend  these  proceedings 
to  your  attention.  To  the  participants — continue  the  dialogue. 


Walter  W.  Hollis 

Deputy  Under  Secretary  of  the  Army 
(Operations  Research) 


WORKSHOP  SUMMARY 


The  Department  of  Defense  has  embarked  on  a  modernization  program  unprece¬ 
dented  since  World  War  II.  The  sophistication  and  complexity  of  emerging 
weapon  systems  have  rendered  managerial  as  well  as  technical  decisions  exceed¬ 
ingly  difficult.  As  technological  boundaries  have  expanded,  the  attendant 
risk  and  uncertainty  of  defense  acquisition  programs  have  also  expanded. 
Morever,  in  this  fast-paced  environment,  the  state-of-the-art  in  managing  risk 
and  uncertainty  has  not  kept  pace  with  the  state-of-the-art  in  technology. 
Often  the  result  has  been  cost  overruns  and  schedule  slippages. 

The  first  step  in  managing  risk  and  uncertainty  is  to  admit  that  they  exist. 
This  has  been  partially  accomplished  and  is  evidenced  by  nunerous  references  to 
risk  and  uncertainty  in  DOD  literature.  In  1969,  Deputy  Secretary  of  Defense, 
David  Packard,  wrote  a  memorandum  to  the  military  services  in  which  he  listed 
inadequate  risk  assessment  as  one  of  five  major  problem  areas  in  system  acquisi¬ 
tion.  Since  that  time,  slow  but  steady  progress  has  been  made.  DOD  Directive 
5000.1  and  DOD  Instruction  5000.2,  the  top  acquisition  policy  documents,  have 
gradually  become  more  specific  in  their  requirement  that  risk  analysis  must  be 
included  as  an  integral  part  of  the  acquisition  process.  More  recently,  the 
DOD  Acquisition  Improvement  Program  that  evolved  from  the  1981  Carlucci  Initia¬ 
tives  directed  the  military  services  to  estimate  most-likely  costs  and  budget 
for  risk.  A  result  of  all  this  high  level  recognition  has  been  the  development 
and  implementation  of  such  concepts  as  Decision  Risk  Analysis  (DRA)  and  Total 
Risk  Assessing  Cost  Estimates  (TRACE).  A  basic  legitimacy  has  been  established 
and  a  framework  is  now  in  place  to  permit  development  and  application  of  new 
methodologies. 

This  workshop  is  the  third  gathering  of  the  risk  analysis  community  to 
specifically  address  acquisition.  The  first  was  hosted  in  February  1979  by  the 
University  of  Southern  California  and  was  not  limited  to  the  Defense  environment. 
The  second  was  co-sponsored  in  February  1981  by  USC  and  the  Air  Force  Business 
Research  Management  Center  and  became  more  defense-oriented.  Based  on  the 
success  of  the  two  previous  conferences,  this  third  workshop  was  commissioned 
by  the  Defense  Acquisition  Research  Council  and  hosted  by  the  Army.  The  work¬ 
shop  in  effect  has  evolved  into  a  biannual  Defense  activity. 

The  theme  of  this  year's  Workshop  was  "Management  of  Risk  and  Uncertainty 
in  Systems  Acquisition"  and  the  general  objectives  were  to  report  on  state-of- 
the-art  techniques  and  exchange  information  on  risk  and  uncertainty  in  DOD.  It 
was  called  a  workshop  rather  than  symposium  to  emphasize  an  interactive 
environment  in  which  maximum  participation  and  discussion  were  encouraged. 

The  format  of  the  workshop  was  based  on  two  concurrent  tracks,  one  focusing 
on  Quantitative  Risk  Analysis,  and  the  other  on  Managerial  Use  of  Risk  Analysis. 
Each  track  consisted  of  five  panels,  each  with  a  panel  chairman  and  two  to 
four  presenters.  Additionally,  there  were  three  plenary  sessions  with  presen¬ 
tations  of  general  interest  to  the  entire  body.  At  any  given  time,  there  were 
at  most  two  concurrent  speakers.  This  enabled  the  participants  to  attend  and 
contribute  to  a  variety  of  different  discussions.  Computer  demonstrations  on 
risk  software  were- also  given  by  various  experts  by  appointment. 


The  participants  in  the  workshop  included  representatives  from  each  of 
three  mil itary  services  anri  DOD  staff,  private  industry  and  academia.  Panel 
chairmen,  speakers,  and  general  attendees  were  all  carefully  selected  from 
among  experts  in  the  field.  This  broad  spectrin  of  viewpoints  and  experience 
assured  a  healthy  exchange  of  information. 

The  program  started,  continued,  and  finished  strong  with  outstanding  plenary 
speakers.  After  introductions  by  the  Workshop  Chairman  and  the  Acting  Comman¬ 
dant,  COL  Forburger,  the  opening  address  was  conducted  by  Dr.  Alan  J.  Rowe, 
USC,  who  had  chaired  §nd  co-chaired  panels  in  the  two  previous  Workshops. 
This  was  followed  by  a  plenary  panel  consisting  of  senior  level  decision  makers, 
Mr.  Hollis  of  the  Army,  Dr.  Shoup  of  the  Navy,  and  Mr.  Thomas  of  the  Air  Force, 
who  made  presentations  and  answered  questions  from  the  audience.  To  start  off 
the  second  day,  Mr.  Bowers,  the  Chief  Executive,  Sanders  Associates  and  former 
Assistant  Secretary  of  the  Navy  {Installations  and  Logistics),  delivered  an 
informative  and  provocative  presentation  on  the  sharing  of  risk  between  Govern¬ 
ment  and  Industry.  The  final  two  speakers.  Dr.  Cetron,  from  Forecasting  Inter¬ 
national,  and  Dr.  Hurta,  of  the  Naval  War  College,  rounded  out  the  program. 
Dr.  Cetron  talked  about  the  uncertainties  of  the  future.  Dr.  Hurta  chaired  a 
practical  clinic  on  the  analyst's  difficulties  in  working  with  decision  makers. 


The  ten  workshop  panels  were  where  most  of  the  work  took  place.  Outstanding 
experts  presented  papers  on  a  variety  of  germane  topics,  of  a  practical  as 
well  as  theoretical  nature.  These  sessions  were  informative,  spirited  and  dis¬ 
played  a  great  deal  of  interaction  between  the  participants.  The  two  objec¬ 
tives  of  the  workshop,  to  "report"  and  to  "exchange  information"  were  unques¬ 
tionably  achieved. 


This  workshop  not  only  achieved  its  objectives  but  produced  two  important 
side  benefits.  It  got  representatives  from  the  three  military  services  and  the 
private  sector  talking  to  each  other  in  a  frank  and  open  manner.  And,  it 
surfaced  controversial  issues  that  require  additional  research,  perhaps  for 
the  next  workshop.  The  primary  issues  are  the  need  to  better  translate  theory 
into  application  and  to  break  down  the  barrier  that  exists  between  analysts  and 
decision  makers.  Theory  is  wasted  unless  it  is  translated  into  practical 
application  and  effectively  communicated  to  senior  level  decision  makers. 


Finally,  the  need  for  periodic  workshops  of  this  nature  is  critical  if  we 
are  serious  about  coming  to  grips  with  risk  and  uncertainty  in  systems  acquisi¬ 
tion.  It  is  only  by  talking  to  each  other,  sharing  information,  and  surfacing 
issues  that  we  can  hope  to  advance  the  state-of-the-art. 

ROBERT  F.  WILLIAMS,  Ph.D. 

Chairman,  Defense  Risk 
and  Uncertainty  Workshop 


fS 

RICHARD  D.  ABEYTA 


Administrative  Chairman, 
Defense  Risk  and 
Uncertainty  Workshop 
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Analysis  &  Evaluation  (Army) 

Weapon  System  Cost  Risk  Reviews 

Richard  J.  Baker 

US  Army  Materiel  Development  and 
Readiness  Command 

Budgeting  for  Technological  Risk  in 
Procurement 


Ivan  A.  Somers 
Hughes  Aircraft 

Causal  Integrative  Model  (CIM) 
1700  Close 


Thursday,  14  July 

0830  Plenary  Paper:  The  Equitable  Sharing  of  Risk  Between  Defense  and 

Industry  (Abilene  Room) 

Jack  L.  Bowers,  CEO,  Sanders  Associates,  Inc. 
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0930  Methods  and  Models  II  (CR  A) 


Chairman,  Kneale  T.  Marshall 
Naval  Postgraduate' School 

Robert  Black 

Grumman  Aerospace  Corporation 
Approximately  Bounded  Cost 
Risk  Regions 


Alan  D.  Kazanowski 
The  Aerospace  Corporation 
A  Quantitative  Methodology 
for  Estimating  Total  System 
Cost  Risk 


Behavior  Under  Risk  and  Uncertainty  (CR  B) 


Chairman,  Robert  F.  Williams 
US  Army  Logistics  Management  Center 

T.  0.  Jacobs 

Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
Cognitive  Correlates  of  Behavior  Under 
Conditions  of  Uncertainty 

Ralph  Swalm 
Syracuse  University 
The  Contribution  of  Cardinal  Utility 
to  Real  Life  Decisions 


Roy  M.  Gulick 

Decisions  and  Designs,  Inc. 

A  Framework  for  Risk  Assessment:  Un¬ 
certainty,  Consequence,  and  Decision 


1045  Coffee 

1100  Split  Sessions  continued 

1200  Lunch 

1300  Risk  Analysis:  Implications 


For  Acquisition  (CR  A) 


Chairman,  William  D.  Rowe 
Institute  for  Risk  Analysis, 
American  University 

William  D.  Rowe 
Institute  for  Risk  Analysis, 
American  University 
Risk  Analysis:  A  Top  Down  View 

John  S.  Gardinier 
US  Coast  Guard 
Successful  Risk  Management: 

A  Concept 


Management  View  of  Acquisition 


Risk  II  (CR  B) 

Chairman,  David  D.  Acker 
Defense  Systems  Management  College 


Lewis  R.  Ireland 
SWL,  Inc. 

AN/SAR-8  Risk  Management  Model 


MAJ  Waldon  R.  Kerns 
Air  Force  Business  Research  Manage¬ 
ment  Center 

(Co-author  CA.nT  Michael  C.  Tankersley) 
Appl  ication  of  Risk  Analysis: 

Response  from  a  Systems  Division 

A.M.  Feiler 
Log/An,  Inc. 

Experiences  and  Lessons  Learned  in 
Project  Risk  Management 


1430  Coffee 


I 

1445  Advanced  Theory  (CR  A) 


Chairman,  Jerome  H.  N.  Selman 
Engineering  Research  Asso¬ 
ciates,  Inc. 

Ronald  R.  Yager 
Machine  Intelligence  Institute 
Iona  College 

A  Linguistic  Approach  to  the 
Representation  of 
Risk  &  Uncertainty  in 
Expert  Systems 

Yacov  Y.  Haimes 
Case-Western  University 
Risk  Management  in  a  Multi- 
Objective  Decision-Making 
Framework 

Alvin  W.  Drake 
Massachusetts  Institute  of 
Technology 

The  Uncertainty  Importance  of 
Components  in  Fault  Tree's 

1645  Close 


Friday,  15  July 

0830  Plenary  Paper:  Uncertainties  of  the  Future  (Abilene  Room) 
Marvin  J.  Cetron,  Forecasting  International 
0930  Plenary  Paper:  Problems  in  Applications 


Donald  W.  Hurta,  Naval  War  College 
1015  Coffee 

1045  Plenary  continued 

1130  Closing  Summary 

Robert  F.  Williams 

U.S.  Army  Logistics  Management  Center 
1200  Close  . 


Issues  in  Risk  and  Uncertainty  (CR  B) 


Chairman,  Richard  D.  Abeyta 

Army  Materiel  Systems  Analysis  Activity 


John  M.  Cockerham 

John  M.  Cockerham  and  Associates,  Inc. 
Implementation  of  Risk  Information  into 
the  POD  Decision-Making  Structure 


Bernard  H.  Rudwick 
Planning  Research  Corporation 
Properly  Including  Risk  when  Evaluating 
System  Alternatives 


Joseph  G.  King 

US  Army  Logistics  Management  Center 
Risk  Analysis  Training  Within  the 
Army:  Current  Status,  Future  Trends 


4 


OPENING  ADDRESS 


HISTORY  OF  RISK  AND  UNCERTAINTY 
RESEARCH  IN  DOD 


* 

* 


HISTORY  OF  RISK  ANO  UNCERTAINTY  RESEARCH 
IN  OOD 
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Unlv.  of  So.  Cal.  Hughes  Aircraft  Co. 

Los  Angeles,  CA.  El  Segundo,  CA. 


‘  INTRODUCTION 

The  Department  of  Defense  has  had  a  continuing  concern 
regarding  the  management  ot  risk  and  uncertainty  and  has 
supported  considerable  ettort  in  an  attempt  to  tind  appropriate 
approaches  to  dealing  with  the  problem.  In  part,  the  ditticulties 
that  arise  are  a  result  ot  the  military's  desire  for  maximum 
performance  which  involves  high  technology,  often  at  an 
advanced  state-ot-the-art.  The  uncertainty  ot  congressional 
funding  and  contractual  constraints  and  the  lack  of  definition 
in  the  introduction  of  new  weapon  systems  also  contribute  to 
program  risk  and  uncertainty.  Since  risk  and  uncertainty  will  not 
disappear,  it  is  requisite  that  the  management  ot  the  weapons 
acquisition  process  address  take  these  factors. 

The  Government,  as  is  evident  by  O. M  B.  circular  A- 109,  identified 
the  need  to  deal  with  risk  end  to  have  alternative  system  designs 
submitted  for  evaluation.  Department  of  Detense  Directive  5000.3 
states  that  test  and  evaluation  should  be  conducted  throughout 
the  acquisition  process  to  assess  and  reduce  acquisition 
risk. 

The  objective  ot  this  paper  Is  to  trace  the  significant  approaches 
that  have  been  taken  in  dealing  with  the  risk  and  uncertainty 
associated  with  military  acquisitions.  From  this  perspective, 
direction  for  future  effort  will  be  identified. 

RESEARCH  TRAIL 

Early  work  in  the  field  ot  risk  and  uncertainty  was  done  primarily 
at  the  RAND  Corporation.  In  addition,  each  ot  the  services 
conducted  studies  on  how  best  to  deal  with  the  problem.  The 
major  efforts  that  were  undertaken  will  be  reviewed  along  with 
an  evaluation  of  their  contributions. 

Cochran  (1)  was  one  ot  the  first  to  recognize  the  combined 
effect  ot  concurrency,  technological  uncertainty,  and  contractural 
urgency  on  risk  and  uncertainty.  He  proposed  an  approach  called 
Disruption  Theory  which  attempted  to  predict  the  degree  of 
uncertainty  In  a  given  program.  As  a  result  of  his  efforts,  the 
first  symposium  on  the  management  of  risk  and  uncertainty  in 
the  acquisition  process  was  held  In  1979  at  the  University  ot 
Southern  Calitornia. 

Major  General  Dewey  Lowe,  at  the  1979  symposium,  reviewed 
an  Air  Force  Study  of  seven  major  aircraft  systems  Including 
the  Bl,  F15,  F 16.  A10,  E3A,  E4  and  EF1 1 1A.  This  study  revealed 
a  consistent  Improvement  In  the  ability  to  control  program  cost 
growth.  Starting  In  the  1950's,  major  system  cost  growth 


exceeded  200%,  while  systems  suffered  trom  high  risk,  poor 
definition,  and  low  visibility.  In  the  60's,  risk  was  moderate, 
definition  was  better,  total  package  procurement  and 
concurrency  was  used,  and  cost  growth  was  between  100% 
and  200%.  The  70’s  witnessed  growth  of  less  than  100%  with 
the  application  ot  prototypes,  change  controls  and  Detense 
System  Acquisition  Review  Council.  When  adjusted  for  intlation, 
the  approximately  90%  growth  rate  ot  the  70's  was  really  a 
30%  rate  in  base-year  dollars.  General  Lowe  estimates  that  as 
much  as  70-80%  ot  cost  overrun  was  caused  by  intlation. 

A  second  symposium  was  held  in  1981  at  the  Air  Force  Academy 
which  dealt  with  defining  the  state-of-the-art  In  the  management 
ot  risk  end  uncertainty.  The  current  workshop  on  Defense  Risk 
and  Uncertainty  at  Defense  Systems  Management  College  Is 
a  recognition  ot  the  continuing  importance  ot  the  subject  to  the 
acquisition  community. 

Finally,  the  current  success  with  a  Causal-  Integrative  Computer 
Model  will  be  U3ed  to  help  Identify  future  directions  that  can 
profitably  be  taken  to  "manage  risk  and  uncertainty" 

BACKGROUND 

Because  of  the  complexity  of  the  acquisition  process,  the 
definition  of  risk  has  tended  to  be  associated  almost  exclusively 
with  the  technology  involved  and  to  ignore  the  management 
aspects  ot  the  problem.  Admiral  Freeman  (2)  stated  that  we 
tend  to  blame  acquisition  problems  on  technical  uncertainty  or 
technical  change;  whereas,  what  is  needed  is  a  manager  who 
is  emotionally  capable  ot  managing  uncertainty  and  who  has 
the  ability  of  taking  the  risk  of  making  decisions  based  on 
judgement  with  access  to  limited  or  suspect  data. 

General  Thurman  (3)  supported  this  position  in  his  statement 
regarding  the  planning  for  uncertainty  and  risk  assessment. 
"Managers  ■  are  expected  to  collect,  weigh  tacts  and 
probabilities,  make  optimal  decisions  and  see  that  they  are 
carried  out.  However,  in  development  projecta,  a  clear  sequence 
is  not  possible  because  ot  the  extended  duretlon,  the  many 
technical  unknowns,  the  continual  discovery  of  new  tacts,  and 
the  changing  constraints  and  pressures."  Professor  Winkler  (4) 
extended  this  further  by  hla  concern  for  the  problem  of  reaching 
agreement  with  respect  to  competitive  and  group  decision 
processes  and  the  role  ot  intormatlon  and  the  Impact  ot 
organization  structures  In  situations  with  multiple  decision 
makers.  Dr.  Kerns  (5)  Identifies  two  distinct  aspects  ot  risk 
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and  uncertainty  as  involving  both  correct  technical  analysis  and 
acceptance  and  implementation  of  the  results.  Given  this 
perspective,  the  complexity  of  the  risk  end  uncertainty  problem 
becomes  apparent. 

A  TAXONOMY  OF  RISK  APPROACHES 

In  order  to  review  profitably  the  developments  in  the  field,  the 
following  four  fectors  are  suggested: 

1.  Estimetion  ■  The  ability  to  forecast,  predict  or  identify 
the  risk  involved. 

Perception  •  The  manager's  skill  in  recognizing  risk 
.and  applying  judgement  to  finding  enswers. 

Impact  -  The  consequences  eppliceble  to  situations 
Involving  risk  and  to  determination  of  the  level  of 
vulnerability. 

4.  Management  -  The  knowledge  end  ettitude  required 
to  implement  risk  taking  decisions. 

Keeney  (6),  in  his  article  on  decision  anelysis,  identifies  a  number 
of  factors  which  contribute  to  the  complexity  of  decision 
problems.  These  include:  multiple  objectives,  difficulty  of 
identifying  good  elternatives,  intangibles,  long-time  horizons, 
many  impacted  groups,  risk  and  uncerteinty,  risks  to  life  and 
limb,  interdisciplinary  aspects,  several  decision  makers,  value 
tradeoffs,  risk  attitude,  and  the  sequentiel  neture  of  decisions. 
He  summarizes  these  fectors  into  four  categories  which  he  states 
are  characteristic  of  complex  decision  problems: 

They  involve  high  stekes 
They  heve  compliceted  structure 
There  ere  no  overall  experts 
There  Is  need  to  justify  the  decisions 

He  concludes  that  at  one  extreme  Intuition  end  Informality  may 
be  used  while  at  the  other  extreme,  models  are  needed  to  capture 
the  full  complexity  of  the  problem. 

If  the  problem  of  risk  end  uncertainty  is  looked  at  in  terms  of 
the  ability  to  manage  and  the  Impect  of  the  decision,  the  following 
two  tebles  can  be  constructed: 


2. 

3. 


TABLE  1.  IMPACT  OF  RISK  ON  PAYOFF 


HIGH 


LEVEL 

OF 

RISK 


LOW 


TABLE  2.  MANAGEMENT  APPROACHES  TO  RIS 


HIGH 


LEVEL 

HIGH  RISK, 

BUT  CAN  BE 
CONTROLLED 

VULNERABLE, 
DIFFICULT  TO 
CONTROL 

OF 

’  RISK 

-  . 

LOW  RISK,- 

LOW  RISK, 

MODERATE 

HIGH 

LOW 

PAYOFF 

PAYOFF 

LOW  HIGH 


USE  OF 

FOREC/STS, 

PERT 

PREDICTIVE 

MODELS, 

SIMULATION 

ESTIMATES 

OR  PRIOR 
EXPERIENCE 

DECISION 

SUPPORT 

SYSTEMS 

LOW  HIGH 


MANAGEMENT  COMPLEXITY 


IMPACT 
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An  examination  of  the  texonomy  of  approaches  to  risk  and 
uncertainty  and  the  two  tables  shown  ebove,  indicates  clearly 
that  additional  research  is  needed  to  integrate  these  factors 
with  the  other  aspects  of  the  acquisition  process  such  as 
contracting,  pricing,  etc.  For  example,  Carlucci  (7)  in  a  memo 
to  secretaries  of  military  departments  ceutions,  "It  makes  little 
sense  to  piece  unreasonable  risk  upon  industry  by  meens  of 
firm  commitments  negotiated  at  high  prices,  but  unenforceable 
in  practice  or  in  fact.” 

DESCRIPTION  OF  THE  ACQUISITION  PROCESS 

Any  description,  of  the  acquisition  process  is,  at  best,  only  a 
static  representation  of  an  extremely  complex  set  of 
interdependent  activities.  The  matrix  shown  in  Figure  1  (a)  is 
suggested  as  e  basis  for  understanding  ceusality.  For  example, 
internal  control  assumes  all  things  are  known  and  controllable 
with  estimates  besed  on  past  date,  prccedures,  designs,  etc. 
The  other  three  categories,  however,  represent  the  reality  in 
major  acquisition  end  are  often  overlooked. 

Although  Figure  1  does  not  reflect  the  dynamics  and  interactions 
that  occur  in  an  on-going  organization,  it  does  illustrates  a  number 
of  key  concepts  that  will  be  developed  In  the  report.  The  linkages 
between  the  four  basic  uncertainty  variables  end  acquisition 


management  help  to  define  the  processes,  or  activities  that 
contribute  to  the  uncertainty  of  acquisition  management.  The 
four  basic  uncertainty  variables  considered  are: 

1.  Internet  Control:  A  measure  of  the  organization's 
ability  to  perform  the  task  requirements. 

2.  Customer  Uncertainty:  The  time  compression, 
concurrency,  or  degree  of  overlap  betwen  phase  of 
development,  and  changes  in  scope. 

3.  Technological  Uncertainty:  A  measure  of  the 
state-of-the-art  end  the  degree  of  interdependency 
among  system  components. 

4.  Environmental  Uncertainty  The  factors  that  cause 
disruption,  delays,  shortages,  failures,  etc.  that  are 
not  under  the  control  of  management  in  the  acquisition 
process. 

The  variebles  define  e  network  of  interdependencies  which 
ultimately  contribute  to  the  uncertainty  and  the  consequent 
problems  in  the  acquisition  process. 

The  categories  shown  in  Figure  t  correspond  to  ones  developed 
by  the  USAF  Academy  Risk  Analysis  Study  Teem  (9)  The 
description  of  the  USAF  categories  is  as  follows: 

Internal  Program  Uncertainty:  Deals  with  the  way  in  which 

the  program  is  organized,  plannad  and  managed. 


UNPREDICTABLE 


LEVEL  OF 
UNCERTAINTY 
INVOLVED  IN 
ACQUISITION 


PREDICT  ASLE 


TECHNOLOGICAL 

ENVIRONMENTAL 

UNCERTAINTY 

UNCERTAINTY 

•  ADVANCE 0  TECHNOLOGY, 

•  UNPREDICTABLE  EVENTS. 

•  INADEQUATE  TASK  DEFINITION, 

•  FAILURES. 

•  CONCURRENCY  OF  DESIGN 

•  DISRUPTIONS. 

AND  PRODUCTION. 

•FORCE  MAJEURE. 

•  DESIGN  CHANGES 

•  regulations 

INTERNAL 

CUSTOMER 

CONTROL 

UNCERTAINTY 

•  PROJECT  CONTROLS. 

•CONTRACTUAL  REQUIREMENTS. 

•  RESOURCE  ALLOCATION. 

•  SCOPE  CHANGES. 

•  SCHEDULES. 

•  TIME  COMPRESSION. 

•  ESTIMATES 

•  STRETCH-OUT. 

•  BUDGETS 

•  CONSTRAINTS. 

•INADEQUATE  FUSING 

CONTROLLABLE  UNCONTROLLABLE 


DECISION-MAKER'S 
ABILITY  TO  CONTROL  ACTIVITIES 


FIGURE  1.  ACQUISITION  UNCERTAINTY 
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Uncerla  inty  of  the  initial  estimate  and  its  impact  on  program 
management.  Uncertainty  in  the  acquisition  strategy  and 
outcome.  Uncertainty  in  resources  needed,  flexibility,  or 
lack  of  contingency  plans.  Competing  demands,  including 
conflict  between  reliability,  vulnerability  and  maintainability 
with  performance  and  operating  costs. 

Technical  Uncertainty:  Covers  the  feasibility  ot  develop  ing 
the  system  a)  all,  including  the  degree  of  technical  difficulty 
Generally  starts  with  an  optimistic  estimate  ot  the 
state-ot-the-arl  and  often  leads  to  a  slippery  technical 
baseline. 

Process  Uncertainty:  Deals  with  the  sensitivity  to  changes 
in  the  extenal  environment  such  as  changes  in  priorities 
or  policies,  the  President's  budget,  congressional  political 
considerations,  etc.  Unavaiiability  ot  funding/resources 
when  needed.  Uncertainty  in  criteria  used  for  changes, 
control,  surveillance,  DSARC  decisions,  etc.  Etfects  of 
Inflation  and  government  regulation. 

Target  Uncertainty:  Covers  the  uncertainty  in  meeting 
performance,  cost  or  schedule  goals  and  determination  ot 
needs.  Uncertainty  in  translating  abstract  needs  Into 
concrete  specitications.  Problem  of  early  estimates  which 
are  seldom  revised. 

Marlin  (to),  in  a  paper  on  the  relationship  between  cost  and 
cost  estimation,  indicates  that  a  cost  estimate  Is  at  best  a 
reasoned  guess  about  a  future  outcome.  The  estimate  requires 
judgment  end  therefore  is  subjective  in  nature.  Furthermore, 
because  cost  estimates  are  probabilistic  in  nature  and  are  valid 
only  as  long  as  the  assumptions  on  which  they  were  based 
remain  the  same,  there  is  need  to  revise  estimates  in 
consideration  of  the  uncertainty  which  exists.  He  points  out  that 
the  level  ot  uncertainty  Is  very  high  in  the  early  |  vases  of  an 
acquisition,  in  part  due  to  the  vagueness  of  system 
specifications.  The  uncertainty  Is  reouced  as  Information  Is 
obtained  from  testing  and  evaluation.  The  conclusion  drawn  is 
that  mistakes  ere  made  using  the  wrong  approach  tor  the  given 
phase  of  the  acquisition  cycle. 

The  cor  fusions  drawn  by  Martin,  are  supported  by  studies  which 
show  that  Initial  estimates  seldom  raflect  the  final  cost.  Davis 
( 1 1)  supports  this  finding  In  his  analysis  of  uncertainty  associated 
with  cost  estimating.  He  contends  that  lack  ot  information 
contributes  to  the  uncertainty  ot  cost  estimates.  He  also 
maintains  that  diffusion  ot  authority  and  responsibility  tor  cost 
estimating  throughout  the  acquisition  cycle  complicates  the 
process.  Furthermore,  short-fuse  requirements  Impose  severe 
time  constraints  and  also  contribute  to  estimating  errors.  He 
concludas  that  there  Is  e  need  for  greater  flexibility  In  the 
acquisition  of  major  systems  In  order  to  cope  with  program 
uncertainties.  Single  point  estimates  do  not  recognize  the  beslc 
uncertainty  Involved  in  cost  and  a  cost  range  would  provide 
greater  flexibility  Basically,  he  argues  that  cost  overrruns  will 
be  a  way  ot  lite  unless  consideration  Is  given  to  the  impect 
of  program  uncertainty  on  estimeting. 

# 

In  a  study  conducted  at  USC  ( 1 2).  alx  major  programs  were 
analyzed  to  determine  the  primary  causes  for  cost  growth. 


schedule  shippage  and  performance  degradation.  Twenty-six 
factora  were  Identitied  as  specifically  contributing  to  coat 
overruns. 

Examination  of  the  data  from  the  many  studies  conducted  reveals 
that  there  are  no  simple  answers  to  reducing  cost,  growth  when 
dealing  with  the  uncertainty  inherent  in  the  acquisition  of 
programs.  It  would  be  unreasonable  to  prevent  advances  in 
atate-ot-the-an  and  changes  in  design  to  meet  requirements  that 
are  determined  during  system  test  and  evaluation,  Furthermore, 
there  will  inevitably  be  some  degree  ot  concurrency  during 
development,  as  well  as  overlap  of  authority  because  ot  multiple 
organizations  involved  in  the  process.  There  is  little  doubt  that 
over-optimism  in  new  designs  leads  to  design  changes  and 
ultimately  to  cost  overruns.  Inflation,  changing  political  and 
customer  influences  and  environmental  catastrophies  will 
continue  to  plague  the  acquisition  process.  Low  bidding,  poor 
or  inappropriate  estimates,  improper  budgeting  and  cost 
control-all  contribute  to  the  problem. 

The  importance  of  uncertainty  as  a  cause  ot  overruns  has  been 
documentad  by  the  RAND  studies  (13),  which  after  over  twenty 
years  ot  studying  complex  development  programs  in  the  U.S. 
and  abroad  concluded  that: 

"High  system  cost  growth  appears  to  arise  primarily 
from  efforts  to  subd".'  difficult  technology  on  highly 
compressed  schedules...  (and  the)  acceptance  of 
optimistic  assumptions  about  the  long-term  predic¬ 
tability  of  technology  and  the  cost  ot  coping  with 
it." 

For  example,  in  describing  system  acquisition  experience,  Perry 
(14),  points  out  that  Initial  estimates  tend  to  be  overly  optimistic 
and  do  not  consider,  or  understate,  technological  difficulties 
actually  encountered  In  program  development.  As  a 
consequence,  these  ditficulties  which  lead  to  Increases  in  total 
program  costs  are  seldom  accounted  lor  early  In  a  program. 
He  found  that,  In  nearly  all  cases,  renegotiated  contracts  were 
much  closer  to  actual  performance  requirements  and  that  this 
was  reflectad  in  adjusted  coats.  Thus,  the  earlier  a  prediction 
of  cost  is  made,  the  greater  the  expectod  uncertainty  of  actual 
cost.  In  general.  In  the  early  conceptualization  stage,  the  required 
technological  advances  and  eventual  system  configuration  are 
poorly  known.  Their  conclusions  concerning  cost  growth  and 
performance  faults  were  that  they  were  principally  due  to 
changes  In  program  acope  end  they  were  outside  of  the 
contractor's  control  These  generally  accounted  tor  the 
difference  between  predicted  cost  of  the  original  program  and 
the  tlnal  cost  of  the  program  as  actually  delivered. 

In  Its  attempt  to  control  cost,  the  DOD  Instituted  the  Design 
to  Cost  Concept  (DTC)  tor  major  weapon  systam  acquisition. 
Although  this  approach  was  an  attempt  to  keep  cosl  within  limits 
that  could  be  echieved  by  a  specified  design,  It  is  now  recognized 
that  frequent  changes  can  undermine  confidence  in  the  process. 
Thus,  flexibility  is  needed  because  ot  the  difficulty  In  estimeting 
major  system  requirements  with  precision.  This  Implies  thet 
performance  parameters  must  be  variable  If  cost  remains 
relatively  fixed. 


■tv. 
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DEFINING  UNCERTAINTY 

Although  uncerteinty  is  defined  es  lack  of  knowledge  about 
specific  effects,  it  also  cen  be  examined  in  terms  of  the  factors 
thet  contribute  to  disruption.  Typical  factors  leading  to  disruption 
ere  shown  in  Table  3. 

TABLE  3.  FACTORS  IN  DISRUPTION 

1.  Delay:  gap  in  carrying  out  e  o:cjrem 

2.  Interruption:  short  term  de'sy 

3.  Stretch-out:  slow  down  of  program 

4.  Interference:. delay  by  other  projects/stoppege 

5.  Redesign  change  scope,  redo  previous  work 

6.  Work  stoppage:  partial  interruption 
7  Interdependencies:  indirect  delays 

8.  Shortages  or  errors:  delays  due  to  rework 

9.  Overlap:  interferences  &  delay  due  to  concurrency 

10.  Redirection  of  effort:  disruptive  effect  of  reorganization 

Risk  end  uncertainty  are  defined  using  classical  probability 
theory  es  well  as  how  they  apply  to  the  acquisition  process. 
Lev  (15),  defines  risk  as  the  condition  where  eech  outcome 
of  the  decision  meker  leeds  to  one  of  e  set  of  possible  specific 
outcomes,  each  occurring  with  a  known  probability.  Uncerteinty 
is  defined  by  Lev  es  the  situation  where  the  probabilities  of 
the  verious  outcomes  are  completely  unknown.  Although  risk 
and  uncerteinty  are  often  used  interchangeably,  they  are  not 
the  same  stete  of  knowledge  in  e  given  situation. 

Peck  and  Scherer's  (16)  comprehensive  analysis  of  the  weapon 
systems  acquisition  process  defined  risk  es  the  level  of 
consequences  of  e  wrong  prediction.  They  operationally  defined 
uncerteinty  es  the  relative  unpredictability  of  en  outcome  of  e 
contemplated  action.  They  categorize  uncertainty  es  either 
internal  or  external  where  internal  uncerteinty  related  to  the 
possible  incidence  of  unforeseen  technical  difficulties  In  the 
development  of  a  specific  weapons  system.  Examples  of  internal 
uncertainty  Include  development  time  of  Interrelated  tech¬ 
nologies.  substitutable  technologies,  and  performance  to 
specification.  External  uncertainty  covers  factors  external  to  a 
given  project,  but  affect  the  course  and  outcome  that  can  be 
expected.  Examples  include  rate  of  technological  change  in 
weaponry,  changes  in  strategic  requirements  and  shifts  In 
government  policy. 

Tha  USAF  Risk  Analysis  report  (9)  defines  risk  as  the  probability 
of  an  occurrence  and  uncertainty  es  incomplete  knowledge.  A 
risk  assessment  is  where  estimates  ere  made  of  the  risk 
associated  with  glvan  alternatives  and  risk  menegament  as  the 
actions  taken  to  reduce  risk.  Risk  analysis  is  considered  the 
combination  of  risk  assessment  end  risk  mengement.  It  is  tha 
letter  definitions  which  are  most  dlractly  eppllcabie  to  the 
acquisition  process.  As  was  shown  previously,  thay  usa 
uncertainty  to  dascriba  target,  technical,  Internal  program  end 
process  effects.  Thay  also  use  a  network  simulation  to  develop 
Individual  and  joint  risk  profiles  es  tha  system  progresses  ovar 
tima. 


Herrison  (17)  defines  risk,  certeinty,  end  uncertainty  as 
follows: 

risk  -  e  common  stete  or  condition  in  decision-meking 
characterized  by  the  possession  of  incomplete  information 
related  to  e  probabilistic  outcome. 

ceitainty  -  an  uncommon  state  of  neture  characterized  by 
the  possession  of  perfect  information  related  to  e  known 
outcome. 

uncertainty  ■  an  uncommon  stete  of  neture  characterized 
by  the  absence  of  any  information  related  to  e  desired 
outcome. 

Harrison  further  contends  thet  "genuine  uncertainty  is  as  common 
es  complete  certainty".  The  more  common  stete  of  nature  is 
incomplete  or  imperfect  information,  which  means  thet  the 
expected  outcome  contains  an  element  of  risk  for  the  decision 
maker.  There  is  no  situation  that  deels  with  the  future  that  can 
be  completely  known  when  the  acquisition  process  lests 
anywhere  from  2  to  12  yeers.  How  can  a  program  manager 
possibly  forecast  events  that  fer  in  the  future  with  any  meaningful 
degree  of  accuracy? 

Beverly  (18),  describes  uncertainty  in  systems  acquisition  as 
the  leek  of  knowledge  in  development  requiring  state-of-the  art 
technology.  Risk,  on  the  other  hend,  is  besed  on  historicel 
phenomena  for  which  probebilities  cen  be  established,  end 
certeinty  or  uncertainty  deals  with  the  existence  of  knowledge. 
Uncerteinty  Is  greatest  when  knowledge  is  at  its  iowest  ievel. 
Uncertainty  would  describe  the  situation  where  e  new  system 
is  being  developed  which  Involved  advanced  state  of  the  art 
technology.  The  lack  of  knowlege,  In  turn.  Inevitably  leeds  to 
errors  in  estimating,  in  design  and  ultimately  In  cost  control. 
This  leeds  to  three  kinds  of  uncertainty  In  weapons  acquisition: 
design  and  technology  uncertainty,  scheduling  uncerteinty  end 
cost  uncertainty.  They  point  out  thet  there  Is  conflict  among 
goals  because  reduced  design/  technology  uncerteinty  enhances 
performance  while  cost  minimization  tends  to  adversely  effect 
both  performance  end  schedule  goels. 

Merlin  (19)  deals  with  uncerteinty  in  terms  of  our  Inebility  to 
predict  the  future  in  the  face  of  unknown  variables.  His  taxonomy 
of  uncertainty  conditions  represents  e  comprehensive  treetment 
of  the  subject.  He  Includes  fou  beslc  categories  of  uncartelnty 
as  follows: 

1.  Environmental: 

a)  natural  factors 

b)  social  &  political  effects 

c)  communication  disperitles 

d)  time  which  results  In  distortions 

e)  external  to  the  project  or  axoganous 

f)  Intarnel  approaches  or  endogenous 

2.  Functional: 

a)  Income  ’business  risk 

b)  financial/ eernlngs  risk 
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c)  tachnological  uncertainty 

d)  production  inadequacies 

3.  Informational: 

a)  unknowns  of  which  contractor  is  aware 

b)  unknowns  that  cannot  ba  foreseen 

c)  lack  of  knowledge 

d)  unknowns  that  cannot  be  anticipated 

4.  Technical: 

a)  unce'tainty.-  no  known  probability  distribution 
of  events 

d)  risk  -  outcomes  can  be  described  by  a  probability 
distribution 

c)  cartainty  -  predictable  outcome  determined 

d)  subjective  ■  probabilitias  derived  independent 
of  tha  problem  at  hand 

Martin  describes  a  twenty  year  period  in  which  measures  to 
reduce  cost  growth  were  not  effective.  He  recommends  the 
use  of  antropy  to  measure  tha  lavel  of  information  in  a  system 
which  is  directly  ralated  to  the  uncertainty  undar  which  decisions 
hava  to  be  mada.  As  entropy  incraasas,  so  does  uncertainty 
and  what  is  needed  is  a  means  to  incraase  information  efficacy 
rather  than  increase  choices  or  randomness. 

McNichols  (20)  presents  a  means  for  estimating  tha  distribution 
of  cost  uncartainty  where  actual  costs  diffar  significantly  from 
original  cost  astimatas.  He  contends  that  cost  overrun  is  a 
meaningless  concapt  becausa  all  cost  astimates  raly  heavily 
on  subjective  judgments  and  ara  subject  to  considerable 
uncertainty.  Ha  considars  four  basic  steps  in  the  treatment  of 
uncertainty.  Thase  includa;  ganeration  of  probability  distributions 
for  individual  cost  alements;  generation  of  a  total  cost  by  additive 
distributions;  combination  of  the  probability  density  functions  to 
form  a  compound  distribution;  finally,  the  correlation  or 
dependence  batwaen  cost  elemants  is  takan  into  account.  Tha 
problam  of  uncertainty  then  Is  to  determine  a  measura  of  the 
dagrae  of  difficulty  or  likelihood  of  achieving  cost  goals. 

Tha  dascriptlons  of  risk  and  uncertainty  prosantad  abova 
Illustrate  the  variety  of  approaches  that  can  bj  takan.  The 
ralevenf  question,  however.  Is  how  best  can  management  daal 
with  the  problem  of  uncertainty  In  the  acquisition  process. 
Although  it  Is  assumed  commonly  that  any  major  overrun  signifies 
poor  managemant,  this  pramlse  falls  to  recognize  that  uncertainty 
Is  Inherant  in  acquisition  and  that  managers  operate  under  severe 
time  and  resource  constraints. 

TECHNOLOGICAL  UNCERTAINTY 

Technological  uncertainty  refers  to  either  the  highly  abstruse 
demands  at  the  very  forefront  of  scientific  knowledge  or  the 
major  gap  between  an  organization's  area  of  expertise  and  what 
Is  required  to  perlorm  effectively. 

In  order  to  examine  technologicaf  advance,  factors  are  needed 
to  determine  the  state-of-  the-art.  The  ones  shown  In  Table 
4  provide  a  atartlng  point  (8): 


TABLE  4.  FACTORS  THAT  CAN  BE  USED  TO  DETERMINE 
STATE-OF-THE-ART 

1.  Size  -  number  of  interrelated  components,  physical 
voluma 

2.  Complexity  •  difficulty  in  meeting  performance 
requirement 

3.  Experimental  nature  of  technology  -  has  it  been 
proven 

4.  Degree  of  newness  •  percent  of  components  of  proven 
tachnology 

5.  Company’s  experience  in  the  field  •  work  on  similar 
programs 

6.  Interdapandancy  of  subsystems  -  number  of  linkages 

7.  Dagree  of  precision  •  quality  requirements 

8.  Unique  resources  -  testing,  or  tooling  requirements 

9.  Dafinitiva  specifications  -  clarity  in  meeting 
requirements 

10.  Dasign  flexibility  -  tolerance  level,  substitutes  available 

It.  Required  theoretical  analysis  -  need  to  support  proposed 
dasign 

12.  Degree  difference  from  existing  technology  -  lifa  cycle  of 
technology 

■13.  Infra-structure  support  required  -  dagree  of  dependency 
on  vendors 

The  factors  shown  in  Table  4  include  the  newness  as  well  as 
the  dasign  requiramants  for  datermining  the  state-of-the-art. 
Thus,  stata-of-thaart  for  a  given  organization  can  be  construad 
as  the  "ability”  to  produce  a  given  dasign,  in  addition  to  the 
nawnass  of  the  technology  involved. 

Technological  uncertainty  also  arises  from  (he  overlap  or 
"concurrance"  of  davalopment  and  production.  The  perceived 
necessity  to  Initials  tha  pondarous  and  involved  processes  of 
production  befora  thare  is  certainty  as  to  tha  stability  of  tha 
product  design,  placas  programs  at  the  mercy  of  changes  which 
occur  in  the  dasign.  Such  dalays  or  changes  are  more  likely 
to  occur  as  the  degree  of  concurrancy  Increasas. 

In  ragard  to  technological  uncertainty,  Duvivier  (21)  recommends 
tha  usa  of  tachnological  forecasting  to  assess  tha  risk  In  meeting 
tha  demand  for  Increasingly  advancad  tachnology.  Ha  postulates 
that  advances  are  extrapolations  of  current  knowledge  and  that 
breakthroughs  ara  rare.  Even  whan  breakthroughs  do  occur, 
such  as  the  laser,  it  takes  8  to  t2  years  to  incorporate  them 
In  new  syslems.  He  shows  examplas  of  engine  weight,  lift  and 
fual  consumption  all  following  smooth  curves.  Thus,  the  cost 
and  benefit  of  new  technologies  can  ba  basad  on  an  extrapolation 
of  technology  growth  curves. 

Because  technological  uncertainty  Impacts  projects  with 
advanced  state-of-the-art,  reduction  In  development  time  Is 
possible  through  the  malntainance  of  a  strong  research  and 
development  posture  New  technologies  can  be  tested  and 
evaluated  prior  to  Incorporation  In  major  systems  and  thus 
“avoid"  some  of  the  uncertainty.  Considering  that  new 


tecnnology  is  limited  to  a  small  percent  of  components,  advanced 
or  anticipatory  development  can  contribute  significantly  to  the 
reduction  of  technological  uncertainty,  reduced  need  for 
concurrency  and,  ultimately,  reduced  disruption.  Thus, 
"demonstreted"  technical  capability  could  supplement  "fly 
before  buy"  as  an  approach  to  the  management  of  risk  and 
uncertainty  in  major  acquisitions. 

If  the  degree  of  state-of-the-art  Is  a  driver  of  technological 
uncertainty,  then  interrelatendness  is  a  major  multiplier  on  cost 
of  development-  and  production.  Interrelatedness  of  design 
relates  a  change  in  one  component  or  subsystem  to  many  others. 
Interrelatedness  can  also  affect  production  and  vendoractivities, 
since  a  change  in  production  methods  or  delivery  cycle  in  one 
erea  or  component  may  affect  production  of  other  components 
or  work  in  other  areas.  A  product  in  an  advanced  area  of 
technology  will  be  subject  to  hiqner  levels  of 
interrelatedness 

INTERNAL  CONTROL 

Internal  control  defines  the  level  or  degree  of  unknowns  that 
are  internal  to  the  system  rather  than  the  external  exigencies. 
Factors  related  to  internal  uncertainty  could  be  measured  using 
dimensions  such  as: 


SERIAL  overlap 

CUSTOMER  DEMAND  REPRESENTED 
BY  DEGREE  OF  CONCURRENCY 


FIGURE  2  IMPACT  OF  CONCURRENCY  ON  DISRUPTION 
AND  RESPONSE  CAPACITY 


1.  The  organization's  ability  to  respond  to  new  or 
unforeseen  requirements. 

2.  The  sleek  or  flexibility  that  has  been  built  into  the 
orgenizetion. 

3.  Prior  experience  with  the  given  technology. 

4.  Number  of  linkages  of  subsystem  dependencies  or 
interaction  with  other  projects. 

5.  Percent  of  the  project's  subsystems  being  developed 
that  are  at  the  state-of-the-art  of  the  technology. 

6.  The  amount  of  time  compression  or  tightness  of 
schedules  (concurrency). 

7.  Availability  of,  or  access  to,  resources. 

8.  Maturity  in  the  planning  and  control  of  operations, 
including  computer  systems  and  organization 
structure. 

9.  Amount  of  overlap  of  development,  design,  and 
Implementation. 

10.  Number  of  contractors  or  organizations  involved  in 
the  project. 

These  factors  contribute  to  manegement’s  ability  to  cope  with 
uncertainty.  In  turn,  the  delay,  disruption,  or  slippage  that  can 
be  anticipated  would  be  measured  by  the  relationship  of  this 
capacity  to  customer  demand  as  shown  In  Figure  2  (8). 
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Expected  disruption  is  an  exponentially  increasing  function  which 
is  dependent  on  the  organizational  response  capacity,  which 
in  turn  depends  on  the  level  of  concurrency.  Thus,  when  the 
level  of  concurrency  approaches  response  capacity,  the  delay 
Increases.  This  formulation  does  not  deal  with  uncertainty  per-se. 
but  whether  the  organization  is  able  to  cope  with  problems  as 
they  arise,  or  is  able  to  cnf'ctpafe  problems.  In  turn,  the  amount 
of  slack  or  flexibility  In  the  organization  defarmines  the  ability 
to  respond  to  uncertain  requiramenls.  If  management  is  operating 
with  minimum  slack,  then  any  disruption  can  cause  a  large 
deiay. 

Another  perspective  on  management  practices  is  based  on  four 
agencies  studied  by  RAND  (22)  covering  R4D  management.  An 
examination  of  the  findings  reveals  the  considerable  latitude 
given  program  managers  in  daaiing  with  creative  individuals 
naeded  In  R&O  programs.  Given  this  kind  of  organizational 
environment,  the  accuracy  of  estimates  is  highly  questionable. 
At  bast,  the  estimate  is  a  fargal  that  permits  a  level  of  effort 
to  be  applied  in  attempting  to  achieve  what  are  often  elusive 
objectives  or  requirements. 

Perry  (13),  in  a  study  of  acquisition  strategies,  recommended 
that  acquisition  mar  agemenf  use  an  incremental  approach.  This 
support  was  basad  in  an  analysis  of  36  major  000  programs 
which  revealed  that  high  cost  growth  was  caused  by: 

1.  Willingness  to  pay  the  price  for  having  high  technology 
with  compiessed  schedules. 

2.  Over-optimism  regarding  the  cost  ot  coping  with  long 
(arm  technology. 

3.  Little  evidence  that  the  programs  had  extreme 
urgency. 

4.  Little  Improvement  In  cost  based  on: 

a)  contractual  approaches 

b)  complex  management  reforms 

c)  Improved  estimating 

d)  early  Identification  end  correction  of  cost 
growth. 

Despite  these  four  factors,  a  number  of  programs  had  surprisingly 
good  oufcomas  and  ware  able  to  predict  cost  performance  and 
schedule.  Using  thalr  findings,  the  authors  suggested  that  an 
Incremental  strategy  end  control  In  the  early  phases  of 
development  would  heve  the  most  effect  on  avoiding  cost 
growth. 

The  Incremental  strategy  recommended  the  following  steps: 

1.  Resolve  uncertainty  earty  In  the  program. 

2  Avoid  concurrency  of  development  and 
production. 


3.  Separate  performance  from  reliability  and 
maintainability. 

4.  Require  periodic  reassessment,  redefinition,  and 
readjustment  regarding  proposed  changes. 

5.  Conduct  tradeoff  studies  to  reeolve  restructuring. 

The  benefits  Irom  an  Incremental  approach  to  management  would 
lie  in  greater  predictability  based  on  prototype  demonstration 
and  in  uncovering  difficulties  early  in  program  life.  It  would  also 
encourage  competition  and  transfer  ot  technology  as  the  need 
required. 

DETERMINING  A  PATTERN  OF  DISRUPTION  AND 
UNCERTAINTY 

Tha  ability  to  detine  causal  reiations  among  variables  in  disruption 
and  uncertainty  is  a  first  step  in  predicting  cost  overruns  and 
in  detarmining  which  actions  a  program  manager  should  take 
to  avoid  cost  growth.  For  example,  Augustine  (23)  proposed 
using  additional  pianning  tunds  based  on  an  assessment  ot  risk. 
He  contends  that  even  the  most  capable  program  manager  is 
not  able  to  forecast  all  the  problems  that  will  be  encountered 
in  a  daveiopment  program  spanning  anywhere  up  to  ten  years. 
However,  It  Is  quite  possible  to  forecast  the  "probability"  that 
additional  funds  will  be  required.  He  recommended  the  use  ot 
TRACE  (Total  Risk  Assessing  Cost  Estimate)  as  the  basis  lor 
justifying  the  additional  (unding. 

Ona  of  the  early  attempts  to  deal  with  uncertainty  was  proposed 
by  Marshak,  Glennon  and  Summers  (24).  They  indicatad  that 
where  "component"  interrelatedness  Is  defined,  one  can  predict 
the  effects  that  are  likely  to  occur.  Under  conditions  of 
uncertainty,  low  slack  heightens  interrelatedness  and 
substantially  increases  the  risk  ot  redesign.  Furthermore,  the 
risk  ot  redesign  Is  sensitive  to  the  degree  that  design  reaches 
beyond  past  state-of-the-art  and  where  there  are  requirements 
to  use  existing  components  which  cen  strain  the  designer  end 
lead  to  suboptimization.  Based  on  three  conditions  describing 
component  interrelatedness,  one  Is  In  a  position  to  predict 
potential  disruption.  When  there  Is  a  high  degree  ot  close 
coupling  or  interrelelednese,  the  .likelihood  ot  design  change 
is  substantial.  Where  there  is  loose  coupling  and  engineering 
slack,  when  components  ere  redesigned  the  deviation  does  not 
Influence  the  other  components,  end  there  Is  less  propensity 
to  redesign.  It  ie  argued  that  the  tightness  ot  component 
interrelatedness  cen  be  traded  off  egelnst  uncertainty,  and  thus 
achieve  more  ettectlve  control. 

Another  measure  ot  uncertainty  Is  system  complexity.  Table 
S  Illustrates  the  Impact  ot  complexity  on  meinteinebility  and 
availability.  Complexity  Is  indicative  ot  the  uncertainty  related 
to  potential  disorder  and  resultant  cost  overruns. 


TABLE  5  COMPLEXITY,  MISSION  CAPABILITY,  AND 
MAINTAINABILITY  OF  VARIOUS 
WEAPON  SYSTEMS. 


(Source  Armed  Forces  Journel  Internetionel,  Mey  1980) 


Mean  Flight 


Degree 

Not 

Hours 

Maintenance 

ot 

mission 

between 

man-hours 

Complexity 

capable 

leilure 

per  sortie 

Air  Force 

A-10. 

low 

32.6 

1.2 

18.4 

A-7D 

medium 

38.6 

0.9 

23.8 

F-4E 

medium 

34.1 

0.4 

38.0 

F-15 

high 

44.3 

0.5 

33.6 

F-111F 

high 

36  9 

0.3 

74.7 

F- 1 1 1D 

high 

65.6 

0.2 

98.4 

Navy/Marine 

Corps 

A-4M 

low 

27.7 

0.7 

28.5 

AV-8A 

low 

39.7 

0.4 

43.5 

A-7E 

medium 

36.7 

0.4 

53.0 

F-4J 

medium 

34.2 

0.3 

82.7 

A-6E 

high 

39.3 

0.3 

71.3 

F-14A 

high 

47.1 

0.3 

97.8 
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FIGURE  3. 


RISK  MODELS 


Meny  approaches  utilize  risk,  rether  then  uncerteinty  to  predict 
possible  outcomes.  Figure  3,  on  the  other  hand,  shows  the 
relationship  between  risk  and  uncerteinty  es  releted  to  causelfty. 
Models  of  known  phenomena  provide  e  more  certain  basis  for 
prediction -then  random  events  which  are  used  for  estimating 
probabilities.  Uncerteinty,  covers  those  areas  thet  are  ill-delined 
or  where  there  is  a  lack  of  knowledge  of  elfects. 

Figure  4  relates  stete-of-the-art  to  interdependency  and  level 
of  concurrency.  The  likelihood  ol  disruption  is  shown  as  a  function 
of  varying  levels  of  concurrency  The  more  complex  the  program, 
end  the  higher  the  interdependencies,  the  greeter  the  likelihood 
of  disruption.  Thus,  the  likelihood  of  disruption  increases  with 
increasing  concurrency  (25). 

CURRENT  APPROACHES  TO  ACQUISITION  MODELING 

A  number  of  representative  models  applied  to  the  acquisition 
of  mejor  systems  ere  examined  here.  The  types  of  models  will 
be  grouped  into  two  mejor  categories  -  probabilistic/stochastic 
models  and  general  models.  Within  this  framework  several 
aspects  of  eech  of  the  models  will  be  explored. 


i 
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SIMPLE 


(LOW  IMPACT) 


(HIGH  IMPACT) 


INTERDEPENDENCIES 
IN  COMPONENTS 


FIGURE  4.  RELATIONSHIP  BETWEEN  STATE-OF-THE-ART 
AND  CONCURRENCY 


The  extension  ol  the  two  groups  of  models  leeds  to  a  proposed 
approach  -  the  Ceusel-  Integrative  Model  (CIM)  -  which  Is 
suggested  as  e  means  to  deel  with  (actors  those  used  by  many 
ol  the  current  models. 

Stochestlc/Probeblllstlc  Models 

Within  this  category,  three  models  will  be  discussed.  These 
are  -  VERT:  the  Risk  Analysis  Model  presented  by  Admiral 
Freeman  et  the  1979  Symposium  on  Risk  end  Uncertainty:  end 
TRACE. 

VERT  -  Venture  Evaluation  end  Review  Technique 

VERT  was  developed  in  1973  and  has  been  used  almost 
exclusively  by  U.S.  Army  program  managers  to  determine  the 
“best"  balance  among  the  three  program  parameters  cost, 
schedule,  and  performance.  The  model  evolved  trom  earlier 
methodological  approaches  such  as  GERT  (Graphical  Evaluation 
and  Review  technique).  CPM  (Criticel  Path  Method),  PERT 
(Program  Evaluation  and  Review  Technique),  MATHNET 
(Mathematical  Network  Analyzer),  and  RiSCA  (Risk  Information 
System  and  Cost  Analysis).  The  short  comings  ol  these  earlier 
models  when  compared  with  VERT  was  their  failure  to  include 
the  performance  variables  along  with  the  cost  and  schedule 
variables  In  the  total  risk-analysis  methodology.  The  VERT  model 
corrects  this  problem. 

A  Monte  Certo  simulation  process  is  It  ated  as  many  times 
as  the  need  warrants  In  order  to  create  a  large  sample'of  possible 


outcomes  concerning:  sleek  lime,  completion  lime.  cost,  end 
performance.  Frequency  distributions,  scetter  diegrems.  end 
probabilities  ol  exceeding  given  values  are  also  generated 
Finally,  pictorial  histograms  are  generated  for  desired  events, 
giving  the  program  manager  an  integrated  risk  enelysis  lor  e 
particular  point  ol  Interest  In  the  program.  Menn  (26)  reported 
in  The  Defense  Management  Journal  that  "some  minor  problems 
have  arisen  with  VERT,  but  none  ere  considered  mejor  obstacles 
to  Its  ettectlve  use."  The  reported  problems  center  about  the 
probability  distributions.  Most  dels  sets  In  VERT  are  triangular 
Indicating  pessimistic,  optimistic,  end  most  likely  values.  This 
(actor  reduces  the  flexibility  ol  the  model  and  the  accuracy  ol 
the  simulations.  Another  problem,  according  to  Mann.  Is  the 
Inability  to  obtain  expert  estimates  ol  the  time  snd  coal 
requirements.  The  experience  is  tha.  most  ol  the  values  obtained 
have  been  overly  optimistic. 

Rlak  Analyala  Model 

RADM  Freemen's  risk  analysis  model  (2)  allows  various 
alternatives  or  systems  to  be  objectively  compared  through 
aggregate  risk  analysis.  The  process  begins  with  a  segmentation 
ol  the  various  program  (unctions  Into  cstegories  redacting  the 
schedule,  cost,  and  performance  variables.  Risk  distributions, 
represented  by  utility  functions,  are  used  to  determine  utility 
values  versus  a  change  In  one  ol  the  variables.  For  example, 
tha  question  ot  "how  much  additional  risk  Is  presented  by  a 
change  In  performance  variable  A  7"  Is  answered.  The  next 
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atep  consists  o(  developing  a  Risk  Matrix  where  the  options 
(or  alternative  systems)  are  presented  versus  the  criteria  for 
choice.  The  summary  risk  or  probability  for  aach 
system /alternative  can  then  be  compared  on  a  quantitative 
basis.  The  term  risk  lactor  is  presented  in  the  form  of  an 
equation: 

Rf  -  1  -  Ps  O  -  C,) 

Where:  R(  —  Risk  Factor 

Ps  —  Probability  of  Success 
C(  —  Consequences  of  Failure 
With:  0  <PS  <1 

0  <C,  <1 

If  C  ,  the  consequence  of  failure,  is  interpreted  to  represent 
a  utility  function,  then  the  risk  factor  curve  will  be  defined  as 
a  utility  function.  The  shape  of  this  function  will  be  in  the  form 
of  a  negative  Pareto  curve.  If  the  system  criteria  and  associated 
risks  developed  from  the  Risk  Matrix  earlier  in  the  sequence 
were  plotted  in  rank-ordered  fashion,  it  too  would  be 
representative  of  a  negative  Pareto  function. 

TRACE  -  Total  Risk  Assessing  Coat  Estimates 

TRACE,  an  approach  designed  to  provide  program  managers 
with  a  method  of  costing  for  risk,  was  developed  by  the  Army 
(27)  in  1974.  The  methodology  incorporates  several  levels  of 
estimates  which  are  available  to  the  user.  These  estimates 
represent  different  levels  of  detail  and  complexity.  The  simplest 
approach  for  a  TRACE  estimate  requires  only  a  baseline  estimate 
by  the  program  manager  and  some  estimate  derived  subjectively 
of  the  total  program  risk. 

Starting  with  the  simplified  approach,  the  model  accepts  inputs 
includ'ng  WBS  elements,  subelements,  interaction  among  WBSs, 
and  unit  probabilities  to  derive  more  detailed  risk  estimates. 
Three  additional  leveis  of  risk  estimates  capable  of  being 
generated  the  are  riak  factor  method,  probabalistic  event 
analysis  and  probabalistic  event  modeling.  The  third  and  fourth 
methods  require  highly  skilled  analysts  to  develop  the  data 
inputs. 

General  Models 

This  category  covers  three  types  of  models  •  regression  models, 
parametric  cost  estimation  and  dynamic  modeling. 

Regression  models  have  contributed  significantly  to  the 
understanding  of  causality  in  the  acquisition  process.  For 
example,  Leech  and  Earthrowe  (28)  have  shown  that  the  ratio 
of  actual  costs  to  estimated  expenditures  can  be  predicted 
based  on  a  regression  with  actual  size  of  the  job.  Using  a 
sample  of  64  Job3,  they  developed  a  regression  curve,  where 
r  —  .955  +  .009X,  and  X  “  actual  job  size  In  man  hours.  As 
they  point  out.  In  every  case  a  commitment  was  made  to  the 
customer  based  on  an  Initial  design.  However,  where  the  job 


is  large,  requires  considerable  technical  innovation  and  the 
quantity  ordered  is  small  (no  opportunity  for  learning)  the  design 
and  development  costs  contribute  significantly  to  the  final  cost 
They  recommend  an  investment  portfolio  approach  to  minimize 
tha  risk  associated  with  design  uncertainty. 

Parametric  Coat  Estimation 

Parametric  Cost  Estimation  is  the  primary  costing  methodology 
for  DOD  weapon  system  acquisition.  This  approach  evolved  from 
research  by  the  RAND  Corporation  in  the  late  1950's.  The  basic 
idea  was  to  make  accurate  estimates  of  weapon  system  costs 
at  the  early  stages  of  system  design.  This  approach  uses 
performance  variables  such  as  speed,  weight,  range,  power, 
etc.  to  predict  costs  because  estimates  of  these  parameters 
are  usually  known  early  in  the  design  phase. 

The  estimates  are  based  on  historical  data  of  previous  or  similar 
systems  and  utilize  statistical  relationships  between  cost  and 
the  performance  parameters  of  these  past  or  similar  systems. 
These  statistical  relationships,  called  cost  estimating 
relationships  (CER),  take  the  form  of  an  equation  which  uses 
cost  as  a  function  of  the  performance  variables  and  constant 
coefficients.  McNichols  (20)  describes  the  relations  in  simplified 
format  by: 

C  -  f  (X)  -  f  (X,.  X2 . Xn) 

where  Xj  denotes,  a  performance  parameter.  The  total  cost  would 
then  depend  on  each  of  the  values  ot  X|  using  data  from  similar 
systems.  McNichols  criteria  for  se'ection  of  the  variables  is 
given  by: 

•  The  logical  or  theoretical  relation  of  a  variable  to  cost 
(thus  Implying  that  a  real  dependence  between  cost 
and  the  value  of  the  particular  variable  or  set  of 
variables  exists,  subject  to  some  random  disturbance 
or  uncertainty.) 

•  The  statiatical  significance  of  the  variable’s 
contribution  to  the  explanation  of  cost  (thus  implying 
that  relevant  cost  experience  exists  to  test  and 
calibrate  the  postulated  cost  dependence  -  subject  to 
measurement  uncertainty.) 

•  The  dependence  pattern  of  the  contribution  made  by 
a  variable  to  the  explanation  of  cost  (thus  the  analyst 
must  have  sufficient  confidence  in  the  relationship  that 
he  Is  willing  to  extend  it  to  estimate  a  new  item  -  and 
different  analysts  will  have  different  degrees  of 
confidence). 

There  are  several  adavantages  to  the  parametric  cost  estimation 
approach.  First,  alnce  the  method  consists  of  a  series  of  CER's 
and  requires  aggregation,  it  is  easily  adapted  to  a  computer. 
Output  and  turnaround  for  new  estimates  can  be  obtained  quickly 
when  compared  with  the  detailed  engineering  approach.  Second, 
sensitivity  analysis  Is  easily  performed  using  this  method.  For 
any  change  In  a  given  parameter,  the  corresponding  change 
In  cost  Is  easily  determined.  Third,  coat/benefit  analyses  or 
trade-offa  are  alLo  easy  to  perform.  Fourth,  each  time  a  new 
generation  system  is  estimated,  the  historical  data  base  already 
developed  can  bn  updated  and  used. 


Dynamic  Modeffng 


CauaaHntegratfve  Model  (CIM) 


Computer-based  dynamic  modeling  was  proposed  by  J.W. 
Forrester  in  the  1950's  as  an  approach  to  help  solve  problems 
of  complex,  continous  systems.  A  dynamio  model  is  based  on 
four  factors  that  have  improved  understanding  of  complex 
systems: 

•  The  theory  of  information-feedback  systems. 

•  A  knowledge  of  decision  making  processes. 

•  The  experimental,  approach  to  analysis  of  complex 
systems. 

•  The  digital  computer  as  a  means  to  simulate  realistic 
mathematical  models. 

Expansion  of  the  concepts  presented  by  Forrester  into  an 
acquisition  model  could  contribute  to  a  better  understanding  of 
the  likelihood  of  cost  overrun  and  disruptions.  The  main 
advantage  of  dynamic  simulation  is  that  it  forces  managers  to 
clearly  define  their  decision  making.  This  approach  leads  to 
greater  Insights  Into  the  acquisition  process. 

However,  dynamic  modeling  is  not  without  disadvantages.  Among 
these  are: 

•  In  simulation,  all  relevant  variablea  and  phenomena 
must  be  quantified.  The  reduction  of  all  descriptive 
knowledge  to  quantitative  measures  is  not  always 
valid. 

•  Dynamic  simulation  Is  found  to  be  most  useful  in 
price-quantity  problems,  less  useful  in  organizational- 
design,  and  least-useful  in  ill  defined  external 
problems. 

•  Dynamic  simulation  is  not  easy  to  apply.  It  is  a  complex 
technique  that  needs  considerable  data  and 
knowledgeable  people. 

•  There  are  problems  in  acceptance  of  the  approach 
because  it  is  often  considered  a  research  tool. 


An  extension  of  the  dynamic  modeling  approach  is  described 
as  a  Causal  Integrative  Model.  The  model  shown  in  Figure  5 
(25)  describes  the  processes,  flows,  variables,  feedback  loops, 
delays,  exogenous  variables  and  key  decisions  as  they  are 
related  to  the  four  basic  variables  in  the  acquisition  process 
shown  in  Figure  1.  As  noted  earlier,  acquisition  models  currently 
being  used  do  not  address  all  of  these  variables:  thus,  each 
of  there  models  lacks  some  degree  of  completeness. 

Referring  to  Figure  5,  the  Causal-Integrative  Model  can  be  used, 
for  example  to  determined  how  a  change  in  economic  uncertainty 
affects  the  level  of  environmental  uncertainty  which,  in  turn, 
affects  mission,  scope,  and  funding.  These  changes  pertrub 
the  system  to  effect  changes  in  organizational  slack, 
technological  uncertainty,  and  customer  urgency.  Thus,  a  change 
in  one  variable  can  bo  shown  to  cause  changes  in  the  others 
through  the  pervasive  network  of  interjependencies.  These 
changes  in  a  key  variable  impact  the  acquisition  cycle  in  ways 
that  are  not  intuitively  obvious  without  the  aid  of  a  dynamic  model 
to  deal  with  the  causal  relationships. 

The  direction  in  acquisition  management  prompted  by  this 
approach  requires  the  following: 

•  development  of  a  comprehensive  computer-based 
acquisition  model, 

•  testing  of  the  model  with  actual  programs, 

•  validation  of  the  model  using  current  programs, 

•  implementing  the  model  for  policy  level  decisions  in 
acquisition  management. 
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CONCLUSION 


The  material  presented  hera  has  attempted  to  highlight  research 
related  to  tha  DOD  acquisition  piocess.  Because  of  the 
pervasiveness  of  the  subject,  of  necassity,  not  all  relevant 
research  or  applications  could  be  included.  Rather,  what  has 
baen  prasentad  hera  can  be  considered  as  indicative  of  the 
current  state-of-tha-art  in  acquisition  management  and  a  baseline 
approach  for  future  developments. 

For  example,  it  was  pointed  out  that  uncertainty  and  disruption 
cannot  be  eliminated,  but  rather  can  be  controlled  if  thera  ara 
causal  models  that  relate  cost  to  advances  in 
state-of-the-art. 


The  acquisition  community  has  daveloped  many  programs  that 
incorporate  approaches  to  manage  risk  and  uncertainty.  'rhese 
programs  included  procurement  methods  such  as  Design- 
to-Cost,  Total  Package  Procurement,  VERT,  and  Life  Cycle 
Costs.  Post-acquisition  attempts  to  control  risk  and  uncertainty 
included  PERT  and  Jts  derivatives,  CPM,  C/SCSC,  and  CSSR. 
In  a  study  of  47  major  programs  reported  in  Selected  Acquisition 
Reports  (SARs)  dated  30  Juna  198t,  Brabson  (29)  indicates 
that  current  estimates  for  these  programs  are  2 18%  of  the  original 
estimatas.  These  levels  of  cost  growth  indicate  that  the  problem 
of  risk  and  uncertainty  is  still  impacting  the  acquisition  community 
ragardless  of  the  efforts  employad. 
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ABSTRACT 


This  i  paper  describes  the  evolution  of  the 
Risk  ana  Decision  Analysis  Project  at  Lockheed- 
Georgla  which  develops  new  methods,  automates 
existing  methods,  and  performs  analyses  in 
support  of  program  definition  and  conceptual 

design  of__new _ air _ vehicles.  The  analytical 

methods 'that  we  have^  found  most  useful  are 
identified.  Application  such  as  technology 
assessment  and  program  risk  analysis  are  dis¬ 
cussed.  Observations  are  given  on  the  practice 
of  decision  analysis  in  an  engineering  environ¬ 
ment. 


INTRODUCTION 


Interest  in  decision  analysis  at  Lockheed- 
Georgla  began  as  a  result  of  a  U.  S.  Air  Force 
(USAF)  competition.  On  15  October  1980  Loek- 
heed -Georgia ,  along  with  Boeing  and  McDonnell- 
Douglas,  received  a  Request  for  Proposal  (RFP) 
from  the  Air  Force  (Reference  1)  to  develop  and 
build  a  transport  aircraft  known  as  the  C-X 
(Cargo  Experimental).  The  C-X  was  specified  to 
be  a  dual-purpose  airllfter,  having  both  inter¬ 
continental  range  and  short  field  landing  and 
takeoff  capabilities.  Responses  were  due  on  15 
January  1981. 

Oie  of  the  proposal  voluaes  specified  by  the 
RFP  was  entitled  simply  "Risk."  Lockheed- 
Georgia’s  Engineering  Systems  Analysis  Division 
(Figure  1)  was  assigned  to  select  a  study 
approach,  collect  information,  conduct  a  risk 
analysis,  and  write  a  40  page  volume.  Unlike 
most  proposal  voluaes  which  describe  what  the 
contractor  will  do  if  he  is  selected,  the  Risk 
voluae  was  specified  to  be  a  report  of  the 
results  of  a  thorough,  systematic  analysis. 
One  of  the  authors  of  this  paper  was  assigned 
as  senior  author  of  the  Risk  voluae. 


Near  the  end  of  the  competition,  it  was 
recommended  to  management  that  when  the 
proposal  team  disbanded  and  Systems  Analysis 
reverted  to  requirements  analysis  and  methods 
development,  an  R&D  project  be  established  to 
continue  the  development  of  methods  for  risk 
and  decision  analysis.  (Xjr  recommendation  was 
followed  and  the  Risk  and  Decision  Analysis 
Methodology  project  was  established  with 
funding  for  one-half  man-year  of  methods  work. 
The  rest  of  our  funding  was  to  come  from  the 
projects  we  supported.  This  division  of  effort 


between  methods  and  applications  continues 
today  in  an  environment  which  we  now  briefly 
describe. 

Engineering  Systems  Analysis  Division  is 
one  of  two  divisions  controlled  by  the  Chief 
Advanced  Design  Engineer,  the  other  being 
Advanced  Design  Division.  Systems  Analysis  is 
responsible  for  requirements  definition  and  for 
performance  of  cost  and  effectiveness  evalua¬ 
tions  of  proposed  designs.  Advanced  Design  is 
responsible  for  conceptual  and  preliminary 
design  studies,  for  both  new  and  derivative 
aircraft.  These  two  divisions  are  managed  by  a 
matrix  arrangement:  Each  working  level  engi¬ 
neer  is  assigned  to  an  R&D  activity  (e.g.,  cost 
methods)  and  participates  in  one  or  more  inter¬ 
disciplinary  projects  as  needs  arise. 

PROJECT  OBJECTIVES 

The  long-range  objectives  of  the  Risk  and 
Decision  Analysis  Methodology  project  are 
to  develop  a  comprehensive  framework  of 
decision  science  methods,  and  implement  these 
methods  st  Lockheed-Georgia  to  assist  planners 
and  decisionmakers.  Annual  objectives  are  set 
to  enable  these  two  objectives  to  be  attained. 
These  short-rangs  objectives  specify  products 
(software  or  study  reports)  to  be  produced  by 
year's  end.  Careful  assessment  is  made  to 
assure  adequate  manpower  is  held  in  reserve  for 
applications  assignments,  both  planned  and 
unplanned. 

The  term  "decision  science"  is  used  here  to 
mean  both  a  problem-solving  philosophy  and  a 
collection  of  methods.  The  philosophy  is  to 
use  the  objectives  and  goals  of  the  manager  in 
structuring  the  decision  situation,  to  allow  as 
input  to  the  model  the  judgment  of  the  manager 
and  his  staff,  and  to  give  explicit  considera¬ 
tion  to  the  problem  environment  (timing, 
organisational  factors,  uncertainty,  and  con¬ 
straints).  By  decision  science  methods,  we 
mean  a  collection  of  techniques  which  include, 
but  are  not  limited  to: 

1.  decision  analysis 

a.  decision  trees 

b.  multiattribute  utility  theory 

o.  probability  encoding 

d.  multiple  criteria  decision  models 

e.  decision  making  under  competition 

f.  policy  testing  via  System  Dynamics 
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Figure  2.  Applied  Reseorch  Approach  Employed  by  '  roject 


Probability  Encoding 

Because  moat  variables  in  engineering 
studies  are  continuous  rather  than  discrete, 
the  decision  analyst  must  have  a  method  for 
encoding  subjective  probability  distributions 
for  such  variables.  By  subjective  probability, 
we  mean  the  "Bayesian  approach"  which  Inter¬ 
prets  a  distribution  as  one's  degree  of  belief 
about  the  outcome  of  future  events. 

We  use  a  four-part  questionnaire  (Figure  3) 
to  convert  responses  from  a  specialist  (techni¬ 
cal  or  management)  Into  a  beta  distribution. 
In  the  example  shown,  we  have  asked  a  relia¬ 
bility  engineer  to  estimate  "Effective  Mission 
Capable  (EMC)  Rate"  fcr  an  aircraft  conceptual 
design.  In  giving  us  his  estimate,  he  takes 
into  account  all  reliability  analyses  conducted 
on  the  design,  together  with  his  experience  on 
previous  aircraft  development  programs.  In 
effect,  the  engineer  serves  as  the  data  base 
upon  which  the  estimate  of  probability  is 
based— he  interprets  the  uncertainty  much  in 
the  same  way  as  the  sample  mean  and  variance 


calculated  in  frequency-based  statistics. 
Caution  must  be  applied  In  extracting  a  sub¬ 
jective  probability  distribution:  assumptions 
upon  which  the  distribution  Is  conditioned  must 
be  specified;  the  bias  in  using  a  single  engi¬ 
neer  to  provide  the  estimate  must  be  addressed 
(perhaps  through  the  use  of  a  Delphi  approach). 

In  Figure  R,  four  beta  distributions  are 
displayed.  Each  distribution  was  obtained  from 
the  project  engineer  responsible  for  the  varia¬ 
ble  shown.  The  specific  values  for  end-points 
and  mode  were  removed  for  proprietary  reasons. 
The  Information  displayed  Is  valuable  In  It¬ 
self,  but  the  real  payoff  is  In  using  these 
data  to  calculate  uncertainty  In  aircraft  range 
with  a  given  mission  payload.  The  next  sub¬ 
section  explains  how  this  is  accomplished. 

Monte  Carlo  Simulation 

We  use  Monte  Carlo  simulation  to  generate  a 
sample  distribution  on  an  output  variable  whose 
distribution  Is  not  known  by  using  a  functional 
relationship  between  this  output  variable  and 


26 


Figure  3.  Conversion  of  Questionnaire  Responses  to  Beta 
Distribution 
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input  variables  with  subjectively  estimated 
distributions.  The  process,  also  known  as 
"quantitative  uncertainty  analysis,"  is  shown 
in  Figure  5.  In.  some  cases,  we  calibrate  the 
function  with  the  results  of  prior,  determinis¬ 
tic  analyses  by  means  of  either  an  additive  or 
a  multiplicative  constant.  For  instance,  with 
the  aircraft  range  example,  we  substituted  the 
nominal  values  ot  the  input  variables  into  the 
Breguet  range  equation  and  solved  for  a  multi¬ 
plicative  calibration  constant  that  would  yield 
the  nominal  value  for  range.  Once  the  function 
is  calibrated,  Monte  Carlo  simulation  quickly 
develops  a  'sample  distribution  for  the  output 
variable.  A  sample  mean  and  variance  may  be 
calculated,  and  a  theoretical  distribution  may 
be  fitted.  Figure  6  shows  the  results  from  the 
aircraft  range  example.  The  probabilities 
shown  are  the  kind  of  information  management 
needs.  Note  that  no  one  individual  on  the 
project  could  have  answered  the  question, 
"What's  the  uncertainty  in  this  aircraft's 
range?"  The  value  of  the  distribution  on  range 
is  highlighted  when  comparisons  among  aircraft 
have  to  be  made. 

ftjantitative  uncertainty  analysis  need  not 
be  restricted  to  a  single  equation,  such  as  in 
the  range  example.  Often,  an  entire  model 


(systems  of  equations)  has  been  linked  as  a 
subprogram  to  the  main  Monte  Carlo  simulation 
program.  An  example  would  be  linking  with  the 
USAF  Cost  Oriented  Resources  Estimating  (CORE) 
model  to  quantify  uncertainty  in  airlifter 
fleet  0  A  S  cost.  Furthermore,  at  Lockheed- 
Ceorgia  uncertainty  analysis  processes  are 
being  performed  sequentially  to  conduct  what 
may  be  termed  "large-scale  uncertainty  analy¬ 
sis."  To  illustrate,  consider  Figure  7  which 
depicts  the  data  flow  for  the  Airlift  Fleet 
Cost-Effectiveness  Uncertainty  Estimator 
(AFCUE)  model  (Reference  2). 

In  AFCUE,  uncertainty  analysis  is  repeatedly 
used  to  convert  distributions  on  independent 
variables  into  distributions  on  dependent 
variables.  The  objective  is  to  convert  un¬ 
certainty  in  key  technical  aircraft  variables 
into  uncertainty  in  fleet  life  cycle  cost.  By 
performing  this  process  on  aircraft  with  dif¬ 
ferent  technology  mixes,  the  effects  of  tech¬ 
nological  uncertainty  on  typical  "point-value" 
estimates  in  conceptual  design  may  be  judged. 
Advanced  technologies,  while  offering  signifi¬ 
cant  performance  benefits,  also  introduce  risk 
into  all  estimates  of  performance,  effective¬ 
ness,  and  cost.  AFCUE  quantifies  this  risk,  in 


•  BUILD  SAMPLE 
ON  VARIABLE  Y 

•  CALCULATE 
SAMPLE  MEAN 
AND  VARIANCE 

•  FIT 

DISTRIBUTION 
TO  SAMPLE 

•  PLOT  SAMPLE 
AND  FITTED 
DISTRIBUTION 
CUMULATIVE 
FREQUENCY 


Figure  5.  Flow  Diagram  for  Quantitative  Uncertainty  Anolysi* 
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effect  layering  an  uncertainty  study  over  the 
standard  conceptual  design  process.  This  seems 
reasonable,  since  the  configuration/technology 
combinations  for  which  cost-effectiveness 
estimates  are  typically  generated  are  10-15 
years  from  production. 

Multiattribute  Utility  Analysis 


historically  resisted  quantification  of  effec¬ 
tiveness  and  worth. 

To  use  multiattribute  methods  to  solve  a 
management  problem,  the  decision  analyst  needs 
five  things: 

1.  A  hierarchy  of  objectives  and  attri¬ 
butes. 


Multiattribute  value  and  utility  methods  are 
receiving  increasing  recognition  and  applica¬ 
tion  in  industry.  At  Lockheed-Georgia ,  multi¬ 
attribute  methods  have  been  used  to  assist  in 
both  program-level  and  design  decisions.  As 
part  of  the  program  risk  analysis  activity  for 
the  C-X  proposal,  we  built  an  additive  multi- 
attribute  utility  model  (Figure  8)  to  represent 
our  customer's  value  system.  This  model  was 
used  to  demonstrate  how  Lockheed's  approach  to 
C-X  risk  areas  minimized  program  risk  (maxi¬ 
mized  expected  utility). 

Two  design  decision  studies  have  used  multi¬ 
attribute  value  models.  The  first  was  a  model 
of  the  C-X  Source  Selection  Criteria,  used  to 
screen  potential  cargo-box  dimension  combina¬ 
tions.  The  second  model  constructed  lends 
structure  to  the  numerous  effectiveness 
criteria  for  a  tactical  airlifter.  This  is 
particularly  significant  since  tactical  air¬ 
lift,  with  its  multiplicity  of  missions,  has 


2.  Characteristics  of  each  alternative  in 
each  of  the  attributes. 

3.  Utility  or  value  functions,  one  for  each 
attribute. 

4.  A  weighting  scheme  for  the  hierarchy 

5.  A  math  model  which  accepts  the  informa¬ 
tion  in  1,  2,  3.  and  4  and  outputs  util¬ 
ity  scores  for  each  alternative. 


The  analyst  must  devote  significant  time  to 
the  first  four  activities  which  by  their  nature 
require  repeated  iteration  and  significant 
personal  interaction.  For  thi3  reason,  the 
availability  of  an  automated  model  (item  5) 
becomes  significant.  The  analyst  need3  to  be 
able  to  (1)  input  the  multiattribute  model  and 
the  characteristics  of  the  alternatives  in  the 
attributes,  and  (2)  have  value,  utility,  or 


Figure  8,  Typical  Program  Mulfiattribute  Model 
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expected  utility  scores  calculated  and  summed 
rapidly.  A  FORTRAN  program  developed  under  RAD 
by  the  project  during  1981  (Reference  3)  ful¬ 
fills  these  two  needs  as  well  as  permitting 
rapid  plotting  of  utility  and  probability 
curves A  flow  diagram  is  shown  in  Figure  9. 

The  preliminary  work  of  defining  criteria 
and  alternatives,  as  described  in  Reference  U, 
is  critical  to  the  success  of  a  multiattribute 
decision  analysis.  These  activities  should 
receive  the  majority  of  the  study  time,  with 
numerical  manipulation  on  the  computer  being 
the  final  step  leading  to  recommendation. 

Critical  Path  Method 

The  use  of  CPM  networks  in  conjunction  with 
project  planning  is  well-known.  Our  experience 
in  Advanced  Design  indicates  that  the  diffic¬ 
ulty  in  pinpointing  staht  and  end  datas  for 
technology  development  projects  means  that  the 
most  useful  data  a  decision  scientist  can  pro¬ 
vide  are:  (1)  how  much  can  a  development 
schedule  be  accelerated  (feasibility),  (2) 
which  activities  should  be  accelerated  (  and  in 
what  order),  and  (3)  what  will  each  increment 
cost.  In  the  literature  this  is  known  as  "time/ 
cost  crashing"  and  the  computer  code  we  use  to 
estimate  the  acceleration  is  aptly  named  CPM/ 
CRASH.  A  flow  diagram  for  CPM/CRASH  is  shown 
in  Figure  10. 


Besides  its  basic  usefulness,  this  program 
is  interesting  for  two  reasons.  First,  the 
program  uses  the  algorithm  described  in 
Reference  5,  nA  Flow-Preserving  Algorithm  for 
the  Time/Cost  Trade-off  Problem"  and  hence 
represents  a  direct  transfer  from  academic 
research  to  industrial  applications.  Second, 
consider  how  CPri/CRAbH  worxs.  Inputs  to  the 
model  ard  (1)  a  .network  description  of  enabling 
activities,  and  (2)  a  linear  time/  cost  trade¬ 
off  curve  for  each  activity.  CPM/CRASH  system¬ 
atically  accelerates  the  development  project, 
"crashing"  the  activity  on  the  current  critical 
path  which  gives  the  maximum  time  reduction  per 
dollar  until  no  further  acceleration  is  possi¬ 
ble.  Zeleny  (Reference  6,  pp  55-58)  points  out 
that  the  project  time/cost  trade-off  curve  is 
an  "efficient  boundary"  in  the  terminology  of 
multiple  criteria  decisionmaking. 

After  obtaining  a  time/cost  curve  for  the 
development  project,  the  decision  analyst  must 
quantify  the  uncertainty  in  the  predicted 
acceleration.  A  confidence  band  about  the 
trade-off  curve  is  needed.  We  obtain  such  a 
band  by  developing  a  probability  distribution 
on  project  completion  time  at  each  of  several 
costs  ranging  from  normal  to  maximum  accelera¬ 
tion.  We  use  network  simulation  rather  than 
the  PERT  formulas  because  of  their  well-known 
underestimation  of  both  mean  and  variance  in 
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Figurts  9.  Flow  Diogrom  for  MulHottribufe  Decision  Analysis 
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CURVES  FOR  ACTIVITIES 

Figure  10.  Process  to  Determine  Project  Time/Cost  Tradeoff  Curve 


Figure  11.  Confidence  Band  on  CPM/CRASH  Results 
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PROJECT  STATUS  AND  DIRECTIONS 


In  this  section,  we  sunnarize  where  we  stand 
in  three  areas:  (1)  development  of  a  unified 
framework  for  risk  analysis  studies  of  DoD 
procurement  programs;  (2)  methods  development 
beyond  the  basic  tools  in  the  preceding  sec¬ 
tion;  and  (3)- applications  to  problems  in  areas 
outside  our  current  environment. 

Program  Risk  Analysis  Approach 

Based  on  two  years  of  methods  development , 
our  current  approach  to  the  types  of  risk 
studies  required  on  new  DoD  business  is  shown 
in  Figure  12.  Our  philosophy  is  that  risk 
emanates  from  the  technical  definition  of  the 
system,  manifesting  itself  in  the  probability 
that  technology,  mission,  cost,  or  schedule 
goals  will  not  be  met.  The  first  two  models 
shown,  QUALM  and  CPM/CRASH,  are  being  used  to 
conduct  the  technology  assessment  task  of  the 
study  contract  "Technology  Alternatives  for 
Airlift  Deployment,"  sponsored  by  USAF  Flight 
Dynamics  Laboratory.  QUALM  is  the  Lockheed- 


Georgia  software  for  implementing  the  quanti¬ 
tative  uncertainty  analysis  process  depicted 
earlier  in  Figure  5.  The  third  model,  called 
CPM/RISK,  is1  an  RAD  task  for  the  project  in 
1983.  This  model  will  take  information  on 
potential  technical  problems,  provided  by  the 
specialties,  and  simulate  the  impact  of  these 
problems  on  a  program  activity  network  with 
both  cost  and  schedule  estimates  on  each  arc. 
Model  output  will  be  probability  distributions 
on  project  time  and  cost,  a3  well  a3  statistics 
for  each  activity  (e.g.,  probability  that  an 
activity  will  be  on  the  project  critical  path) . 
CPM/RISK  is  a  Lockheed-Georgia  modification  of 
a  methodology  conceived  of  by  Kraemer  at 
Boeing-Vertol  (Reference  7). 


Advanced  Tools 


We  are  developing  more  advanced  tool3  than 
those  discussed  earlier.  The  reader  can  find 
adequate  references  in  the  open  literature  for 
the  methods  listed  below.  Each  of  these  we 
consider  necessary  to  make  the  3tep  from  ade¬ 
quate  to  outstanding  methodology  readine33. 
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Figure  12.  Unified  Approach  to  Risk  Analysis 
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Methods  Currently  Heady 


o  Goal  Programming 
o  Decision  Trees 
o  Statistical  Package 

o  Venture  Evaluation  and  Review  Technique 
(VERT) 

o  Impact  Diagram  Method 

o  Iso-Risk  Contour  Method 

Methods  In  Development  or  Ordered 

o  CPM/RISK  Simulation  Model 

o  Graphical  Evaluation  and  Review  Technique 
(GERT) 

o  Linear  Multiobjective  Programming 
o  DYNAMO 

Extended  Project  Environment 

As  described  earlier,  through  1982  the  pro¬ 
ject  applications  had  been  exclusively  to 
activities  within  Lockheed-Georgia's  Engineer¬ 
ing  Branch.  Through  a  series  of  briefings  in 


early  1983.  we  have  reached  out  to  other 
branches  of  the  company,  to  other  divisions  of 
the  corporation,  and  to  potential  DoD  custo¬ 
mers.  The  project  environment  has  therefore 
been  extended  as  shown  in  Figure  13,  resulting 
in  a  respectable  demand  for  our  services.  For 
example,  we  are  currently  engaged  in  studies 
for  two  directorates  which  report  directly  to 
the  president  .  —  Strategic  Planning  and 
Advanced  Programs.  We  also  are  negotiating  a 
government-funded  study  contract  for  1983-89. 
These  requests  for  our  services  simply  reflect 
the  fact  that  managers  in  technologically- 
oriented  work  have  problems  for  which  the 
decision  analyst  can  offer  structuring  and 
quantitative  insight. 

While  methods  development  will  continue  in 
1984,  we  see  our  division  of  effort  shifting 
from  half-time  methods  development  and  half¬ 
time  applications  to  65-70J  applications.  In 
our  view,  we  have  tremendous  advantages  over  an 
outside  consultant  when  working  on  Lockheed- 
Georgia  problems,  even  outside  of  our  immediate 
environment  —  we  are  trusted  with  proprietary 
data,  we  are  aware  of  problem  subtleties  and 
personalities,  and  we  know  tne  organization 
chain,  both  formal  and  informal.  In  summary, 
we  are  becoming  the  "in-house  decision  analysis 
staff"  at  Lockheed-Georgia ,  a  development  which 
Ulvila  and  Brown  forecast  for  all  large  corpo¬ 
rations  in  their  recent  paper  "Decision 
Analysis  Corae3  of  Age,"  Reference  8. 


Figure  13.  Extended  Project  Environment 
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CONCLUSIONS 


REFERENCES 


The  need  for  quantitative  risk  assessment  on 
DoD  technology  studies  and  procurement  programs 
has  been  recognized  since  the  late  60s. 
Methodology  evolved  during  the  70s  to  meet  this 
need,  and  is  now  readily  available.  Methods 
exist  to  assess  uncertainty  in  developing 
technologies  —  uncertainties  in  time  and  cost 
to  reach  maturity,  and  uncertainties  in  opera¬ 
tional  benefit  at  maturity.  Once  technologies 
advance  to  a  point  were  they  may  be  proposed  on 
a  new  aircraft  programs,  methods  exist  to  help 
identify  and  quantify  program  risk. 

An  analyst  equipped  with  methods  as  describ¬ 
ed  in  this  paper  can  contribute  to  the  system/ 
program  definition  and  analysis.  This  contri¬ 
bution,  of  course,  i3  dependent  on  management 
acceptance  and  engineering  specialty  support. 
Management  wants  risk  studies  performed  because 
they  are  acutely  aware  of  uncertainties,  and 
because  their  counterparts  in  the  government 
require  risk  be  identified  and  measured. 
Engineers .will  cooperate  with  the  risk  analyst 
once  they  have  seen  that  their  specialty/judg- 
ment  will  not  be  misrepresented.  The  risk 
analyst  has  become  an  accepted  member  of  con¬ 
ceptual  design  teams  at  Lockheed-Georgia ,  much 
as  the  cost  analyst  did  10-15  years  earlier. 

Risk  analysis  methods  are  treated  as  a 
technology  at  Lockheed-Georgia.  IR4D  funding 
permits  analysts  to  improve  existing  methods, 
to  perform  research,  and  to  create  new  methods 
and  the  accompanying  computer  codes.  This 
funding  has  permitted  us  to  build  an  adequate 
framework  for  risk  analysis  of  defense  systems 
and  programs  in  less  than  three  years.  We  have 
expanded  into  related  decision  sciences  of 
network,  analysis,  statistical  analysis,  and 
decision  analysis.  While  continuing  to  provide 
risk  studies  in  engineering  as  our  primary 
duty,  we  are  now  working  on  a  broader  variety 
of  problems  throughout  the  company. 


1.  Air  Force  Systems  Command,  Aeronautical 

Systems  Division,  Request  for  Proposal  No. 
F33657-80-R-0227,  "Full  Scale  Engineering 
Development/Production:  Interim  Contrac¬ 

tor  Support  and  Spares  for  the  C-X  Air¬ 
craft  System,"  October  1980. 

2.  Batson,  R.  G;,  "Description  of  the  Model 

AFCUE:  Airlift  Fleet  Cost-Effectiveness 

Uncertainty  Estimator  for  Parametric 
Design  Studies,"  Lockheed-Georgia  Company, 
PM92D364-6,  August  1982. 

3.  Bailey,  T.  M. ,  and  R.  G.  Batson,  "Multi¬ 

attribute  Decision  Analysis  Model  (MADAM): 
Program  Description  and  User's  Guide," 
Lockheed-Georgia  Company,  LG81WP7292-002, 
August  1981. 

4.  Keeney,  R.  L, ,  and  H.  Raiffa,  Decisions  with 

Multiple  Objectives:  Preferences  and 
Value  Tradeoffs,  John  Wiley  4  Sons,  New 
York,  1976. 

5.  Tufekci,  S.,  "A  Flow-Preserving  Algorithm 

for  the  Time/Cost  Trade-off  Problem,"  HE 
Transactions,  Vol.  14,  No.  2,  June  1982, 
pp.  109-113. 

6.  Zeleny,  M.,  Multiple  Criteria  Decision 

Making.  McGraw-Hill,  New  York,  1982. 

7.  Kraemer,  G.  T. ,  "A  Successful  Quantitative 

Risk  Assessment  Technique,"  Journal  of 
Parametrlcs ,  Vol.  1,  No.  3,  Winter  19&2, 
pp.  13-21. 

8.  Ulvila,  J.  W. ,  and  R.  V.  Brown,  "Decision 

Analysis  Comes  of  Age,"  Harvard  Business 
Review,  Sep-Oct  1982,  pp.  130-141. 


AD  P00230  1 


John  K.C.  Woo 


Department  of  the  Navy 
NAVSEA  90,  Acquisition  Control  Division 
Washington,  D.C.  20362 


This  paper  presents  an  overview 
of  Naval  Sea  Systems  Command's 
(NAVSEA)  experiences  and  current 
use  of  quantitative  schedule  risk 
analysis  for  major  acquisition, 
ship  overhaul  and  modernization 
programs.  On  February  7,  1983, 
Mr.  Sawyer,  then  A  S  X  ( S  &  L ) 
requested  that  a  quantitative 
analysis  be  provided  of  the  risk 
to  ship  delivery  dates  caused  by 
the  first  introduction  of  a 
selected  new  system.  NAVSEA, 
working  with  NAVMAT,  surveyed  the 
available  schemes  and  current 
quantitative  schedule  risk 
analysis  used  by  NAVSEA  program 
managers  to  determine  if  a 
standard  quantitative  risk 
analysis  scheme  should  be  used 
for  all  Navy  programs.  As  a 
result,  a  number  of  risk  analysis 
predictive  systems  were  looked 
at,  among  them  PROMAP,  TRACE,  and 
SMS.  PROMAP  experiences  and 
lessons  learned  in  project  risk 
management  will  be  presented  in  a 
separate  paper  by  Nr.  A.  Feiler 
during  this  workshop.  TRACE  will 
be  addressed  by  Mr.  Grover,  Nr. 
Alfieri  and  Mr.  Cockerham  during 
this  workshop.  This  paper  will 
describe  very  briefly  the 
computer-based  Schedule 
Management  System  (SMS)  currently 
used  by  several  program  managers 
within  NAVSEA  for  risk  assessment 
as  a  management  tool. 


m£Q££ 

The  purpose  of  this  paper  is  to 
provide  to  other  services  and  to 
other  departments  within  the  Navy 
a  report  on  current  and  planned 
use  of  quantitative  risk 
assessment  by  NAVSEA  as  a 
management  decision  and  reporting 
tool  for  major  acquisition 
programs.  .  Specifically, 
quantitative  risk  assessment/ 
management  as  it  relates  to  the 
acquisition  schedule  is  addressed. 


The  Carlucci  initiative 
(Reference  1)  stated  that  review 
of  47  major  acquisition  programs 
indicated  that  there  was  a  total 
overrun  of  129%  in  final  cost  as 
compared  to  the  milestone  II 
estimates.  Additionally,  this 
report  identified  15%  of  these 
overruns  as  being  attributable  to 
schedule  change.  It  further 
reported  that  of  the  45  programs 
with  schedule  change,  41  had 
increased  costs.  This  is  a 
significant  problem  considering 
the  magnitude  of  these  programs. 
The  important  question  addressed 
here  is  what  could  have  been  done 
to  forewarn  the  manager  of  these 
developing  problems.  In 
countering  the  possibility  of 
cost  overruns  in  major  ship 
acquisition  programs  CHNAVMAT 
recognized  that  managers  must 
take  into  consideration  risk  and 
uncertainty  in  their  decision 
making  processes.  As  a  result, 
on  December  13,  1982  the  MMC 
revised  the  requirements  for 
implementation  of  the  MMC 
Selected  Acquisitions  Tracking 
System  (NSATS)  to  reinforce  the 
requirements  for  program  managers 
to  take  risk  factors  into  account 
within  the  SYSCOMS  (Reference  2). 
In  response  NAVSEA  published 
instructions  updating  NAVSEA 
Policy  and  Procedures  of  the 
revised  NSATS  (Reference  3). 
Under  this  instruction  it  is  the 
policy  cf  the  Commander,  Naval 
Sea  Systems  Command,  that 
acquisition  programs  will  be 
appraised,  the  results  of  which 
will  provide  consistent,  timely 
assessments  to  ensure  the 
programs  are  technically, 
financially,  administratively, 
and  logistically  sound.  These 
assessments  are  to  provide  a  high 
degree  of  confidence  that  the 
programs  are,  in  fact,  sound  and 
executable  and  reflect  the 
position  of  COtlNAVSEA  and  the 
Chief  of  Naval  Material 
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(CBNAVMAT).  .  This  policy  has  been 
implemented  at  four  levelB  within 
NAVSEA.  The  four  level?  are: 

1.  COHNAVSEA  Appraisal. 

COMNAVSEA  will  appraise  those 
acquisition  programs  recom¬ 
mended  by  the  Acquisition 
Review. Board  (ARB),  as  well 
as  others  of  his  choosing. 

2.  NAVSEA  ARB.  Appraisal.  The 
ARB  shall  serve  as. the  prin¬ 
cipal  forum  for  review  of 
acquisition  category  (ACAT) 
programs  which  are  being 
presented  to  the  Office  of 
the  Secretary  of  Defense 
(ACAT  I),  Secretary  of  the 
Navy  (ACAT  IIS),  Chief  of 
Naval  Operations  (ACAT  IIC), 
the  OPNAV  Sponsor  (ACAT  III) 
and  those  being  presented  for 
NAVSEA  approval  (ACAT  IV). 
The  ARB  shall  also  serve  as 
the  principal  forum  for 
program  status  appraisal  of 
designated  programs. 


3.  Directorate  Appraisal. 
Acquisition  programs  which 
are  listed  in  the  NAVSEA 
Acquisition  Program  Index  but 
are  not  appraised  by  the  ARB 
will  be  appraised  annually 
within  the  Directorate  at  a 
level  senior  to  the  Program 
Manager. 

4.  Program-Manager,  Appraisal. 
Each  NAVSEA  acquisition 
program  listed  in  the  NAVSEA 
Acquisition  Program  Index 
will  undergo  Program  Manager 
appraisal  quarterly.  The 
appraisal  will  be  reflected 
in  the  quarterly  Naval 
Material  Command  NSATS  sub¬ 
mission. 


NSATS  REPORTING  REQUIREMENTS 

Figure  1  depicts  the  NSAT  program 
manager  reporting  format. 
Although  selected  programs  within 
NAVSEA  utilize  quantitative  risk 
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analysis  to  meet  program 
objectives,  there  is  no  uniform 
policy  Or  guidance  that  requires 
the  implementation  of  a  standard 
quantitative  risk  assessment 
program/system. 


QUALITATIVE  AND.  QUANTITATIVE 

CQHSID6BAIIQHS 

Figure  1  represents  some  sample 
formats  extracted  from  the  NS AT 
system  which  are  to  be  used  by 
NAVMAT  Acquisition  Managers  for 
the  purposes  of  reporting  their 
status  during  specific  phases  of 
an  acquisition.  Basically,  for 
overall  appraisal,  NS ATS  utilizes 
a  three  point  estimate  -  green, 
yellow  and  red.  Green  indicates 
the  program  element  is 
essentially  on  plan#  yellow 
indicates  there  is  potential  for 
significant  deviation  from  plan, 
and  red  indicates  there  has  been 
a  significant  deviation  from  the 
plan.  These  measures  are 
subjective  in  nature  with  no 
specific  threshholds  established 
to  define  what  is  significant  and 
what  is  not.  It  should  be  noted 
that  the  NS ATS  requirements  do 
not  provide  a  tool  or  method  to 
assess  and/or  control  risk. 
NS ATS  is  a  reporting  format. 

Essentially,  these  reporting 
requirements  of  program  status 
are  qualitative  unless  the 
manager  has  at  his  disposal  the 
tools  which  allow  him  to  quantify 
his  program  status  in  terms  of 
technical,  schedule,  and  cost 
factors.  Only  through  use  of 
such  tools  is  the  manager  able  to 
accurately  measure  progress 
against  the  plan.  A  quantifiable 
methodology  permits  a  clear  and 
measured  definition  of  what  is 
and  what  is  not  significant. 

The  program  manager  in  order  to 
quantify  actual  program  status 
does  not  generally  require 
additional  information.  The 
astute  manager  already  has  the 
preponderant  planning  and 
execution  information  at  his 
disposal.  Each  program  manager 
has  his  own  plan  for  meeting 
program  objectives.  The  plan 
normally  entails  what  has  to  be 
done  (activities),  how  long  it 
takes  to  do  a  particular  activity 
(the*  duration),  how  much  it 
costs,  the  sequence  of  events, 


the  logic  associated  with 
activities  in  the  program,  and 
the  start  and  end  dates.  The 
program  manager  given  this  basic 
data  has  a  strong,  firm  founda¬ 
tion  upon  which  to  add,  at  a 
reasonable  cost  to  him  in  time 
and  money,  the  ability  to  conduct 
quantitative  risk  management  on 
his  program. 

Uncertainty  is  a  key  factor  to  be 
considered  in  the  application  of 
risk  management.  There  are  two 
types  of  uncertainty:  first,  the 
type  most  people  call  ■unknown- 
unknowns*  or  "unk-unks*.  These 
are  unpredictable  variances.  An 
example  of  an  ■unk-unk*  would  be 
damaging  a  component  thus 
requiring  it  to  be  returned  to 
the  factory  for  rebuild.  There 
are  no  systems  which  predict 
■unx-unks"  but  there  are  a  number 
of  variable  factors  that  are  in 
fact  statistically  predictable. 
For  example,  within  an 
organization  standard  operating 
procedures  state  it  will  take  120 
days  from  proposal  receipt  until 
a  contract  is  awarded.  From  a 
procedural  point  of  view  and  from 
a  documentation .  point  of  view 
that's  all  well  and  good; 
however,  a  review  of  the  records 
in  their  contracting  office  may 
show  it  actually  has  taken  from 
90  to  165  days  with  an  expected 
duration  of  120  days.  The  use  of 
the  120  day  figure  alone  would 
induce  error  where  an  estimate  of 
90  to  165  days  with  an  average  cf 
120  days  is  statistically  more 
valid. 

Secondly,  within  an  overall 
program  there  is  predictable 
uncertainty  in  varying  degrees 
related  to  each  individual 
activity,  in  order  to  manage  the 
predictable  uncertainty  and  risk 
in  a  program,  it  is  necessary  to 
determine  those  activities  whose 
uncertainty  must  be  controlled 
and  those  which  do  not  need  to  be 
controlled.  The  list  of  activi¬ 
ties  that  must  be  controlled 
changes  as  progress  is  made  in  a 
program.  The  attributes  of  a 
system  capable  of  handling 
uncertainty  and  risk  are  several. 
First,  such  a  system  must  have  a 
deterministic  base  to  allow  a 
program  manager  to  load  his 
program  plan.  Secondly,  it  must 
have  the  ability  to  handle 
stochastic  or  probabilistic 
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elements  of  uncertainty  in  each 
activity.  Thirdly,  it  must 
facilitate  updating  on  a 
continuing  basis  during  the 
evolution  of  the  program  in  order 
that  progress  can  be  recorded 
against  the  original  plan.  In 
quantifying  progress  vs.  plan  the 
most  important  aspect  is  that  the 
assessment  be  objective  and 
reflect  true  progress.  An 
example  of  the  absence  of  true 
status  assessment,  or  objective 
assessment,  is  assuming'  that  50% 
of  an  activity  is  complete  when 
50%  of  the  manhours  originally 
planned  for  the  activity  have 
been  expended.  In  perpetuating 
this  error  many  industrial 
activities  report  only  the  begin¬ 
ning  of  an  activity  and  the 
completion  of  an  activity  in  a 
truly  objective  way  and  any 
intermediate  progress  is  a 
function  of  the  expenditure  of 
manhours  against  the  plan.  The 
reason  the  objective  assessment 
of  current  status  of  a  program  is 
so  critically  important  is  that 
if  one  waits  until  reported 
completion  (having  missed  a  mile¬ 
stone)  one  has  in  fact  missed  the 
opportunity  to  take  early 
corrective  action.  The  fourth 
attribute  required  is  the  ability 
to  make  predictive  projections  in 
a  way  that  permits  one  to 
determine  whether  or  not  they  are 
still  capable  of  meeting  schedule 
and  what  degree  of  risk  is 
associated  with  achieving  it. 
The  final  attribute  is  the 
ability  to  play  "what  if"  games 
thus  permitting  the  manager  to 
handle  the  effects  of  something 
happening  that  is  not  in  the 
plan,  e.g.,  the  unexpected. 
These  capabilities  must  be  part 
of  a  system  that  is  capable  of 
modelling  the  program  as  the 
program  manager  perceives  it. 
The  manager  must  be  able  to 
tailor  the  system  to  his  specific 
program,  not  vice-versa. 


THE  ORIGIN  QF  QUANTITATIVE  RISK 
MANAGEMENT  IN.  HAYS E A 

The  Critical  Path  Method  (CPM) 
and  Program  Evaluation  and  Review 
Technique  (PERT)  were  developed 
in  the  1950's.  The  strong  point 
of  CPM  was  incorporation  of. 
networking.  PERT  additionally 
accounted  for  estimating  bias. 
Both  required  the  development  of 


a  careful  plan  from  beginning  to 
end.  Prior  to  PERT/CPM,  managers 
had  mainly  used  bar  or  GANTT  type 
charts.  The  ability  to  conduct 
simulations  had  been  available 
since  the  1880's  and  with  the 
advent  of  structured  networking 
and  improved  computer  capabili¬ 
ties  probabilistic  network 
simulation  became  a  reality. 

The  Naval  Sea  Systems  Command 
became  involved  in  risk 
management  with  respect  to  ship¬ 
building  in  the  1960*8.  NAVSEA 
initiated  development  of  a  system 
that  permitted  probabilistic  risk 
analysis  of  ship  acquisition 
projects.  The  product  became 
known  as  PROHAP  and  was  delivered 
to  the  Naval  Sea  Systems  Command 
in  the  early  1970's.  As  a  result 
of  numerous  applications  to 
ongoing  ship  programs  and  lessons 
learned,  there  followed  the 
development  of  the  Schedule 
Management  System  (SMS).  SMS  has 
significant  capabilities  beyond 
those  of  early  PROMAP.  These 
improvements  include  a  capability 
to  compare  probabilistic 
projection  to  plan  in  such  a 
manner  as  to  tell  the  program 
manager  if  his  project  is  on 
schedule;  and  if  not,  what 
management  action  he  can  take  to 
get  it  back  on  schedule. 


DETERMINISTIC.  ANillSQBABILISIIC 
METHODS 

There  are  three  principle  factors 
that  comprise  the  difference 
between  deterministic  and 
probabilistic  analysis.  These 
factors  ares  1)  estimating  bias, 
the  difference  between  the 
estimate  given  by  planners  and 
estimators,  and  the  value 
expected  to  be  realized  after  the 
activity  has  been  completed;  2) 
nodal  bias,  the  phenomenon  that 
occurs  at  each  node  in  a  network 
where  the  completion  distribu¬ 
tions  of  the  predecessor 
activities  overlap;  and  3)  criti¬ 
cality,  the  probability  that  any 
particular  activity  will  be  on 
the  path  that  ultimately  drives 
the  network.  These  three  factors 
coupled  with  network  logic 
generate  three  sets  of  results 
from  the  analysis  of  a  network. 
These  results  are:  1)  the 

schedule  or  planned  dates  for  the 
network;  2)  the  deterministic 


projections  resulting  from  having 
simulated  the  network  determinis¬ 
tically;  and,  3)  the  probabilis¬ 
tic  projections  resulting  from 
having  simulated  the  network 
probabilistically.  Figure  2 
shows  that  these  three  results 
can  occur  in  three  possible 
configurations  on  the  time  line. 
The  first  configuration,  on  the 
left  of  Figure  2,  is  that  in 
which  both  the  deterministic  and 
probabilistic  results  indicate 
that-  the  network  will  complete 
prior  to  the  schedule  date.  This 
particular  condition  has  little 
risk  of  not  completing  on 
schedule,  and  the  program  is 
considered  to  be  in  condition 
green.  This  is  consistent  with 
the  NSATS  condition  green  which 
is  defined  as  being  "essentially 
on  plan".  The  second  configura¬ 
tion  of  these  three  projections 
is  shown  in  the  middle  of  Figure 
2.  In  this  configuration  the 
deterministic  projection 
indicates  that  the  schedule  can 
be  met,  while  the  probabilistic 
projection  shows  that  completion 
is  expected  to  exceed  schedule. 
This  is  defined  as  condition 
yellow  and  is  considered  to  have 
acceptable  risk.  This  is 
consistent  with  the  NSATS  yellow 
condition  which  is  defined  as 


having  a  "potential  significant 
deviation  from  plan"  -  that  is, 
the  deterministic  results  show 
that  in  fact  the  program  can  meet 
its'  schedule  but  the  probabilis¬ 
tic  results  indicate  there  is 
potential  for  exceeding  the 
program  schedule.  The  third 
configuration,  on  the  right  of 
Figure  2,  is  that  in  which  both 
the  deterministic  anu  probabilis¬ 
tic  projections  predict  that 
completion  will  exceed  schedule. 
This  is  defined  as  condition  red 
and  is  considered  to  have 
unacceptable  risk.  This  defini¬ 
tion  is  also  consistent  with  the 
NSATS  definition  of  red  which  is 
defined  as  being  a  “significant 
deviation  from  plan".  In  this 
third  case,  even  if  the  program 
were  to  follow  the  originally 
planned  sequence  and  durations  it 
will  complete  on  the  determinis¬ 
tic  projection,  and  will  have 
experienced  a  significant 
deviation  from  the  original  plan 
which  is  reflected  in  the 
schedule  dates. 


RISK , MANAGEMENT 

A  manager  must  understand  which 
condition  of  risk  his  program  is 
in.  Reference  to  Figure  3  shows 


the  three  program  conditions  of 
risk  at  the  top.  In  condition 
green  the  system  output  is  a 
critical  path  network  and  tabular 
listing  of  each  activity's  start 
and  complete  dates.  The 
management  action  at  this  point 
is  simply  to  monitor  the  program 
because  as  long  as  it  continues 
to  follow  plan  (meet  determinis¬ 
tic  dates),  even  with  the 
uncertainty  aspects  taken  into 
account,  the  schedule  will  be 
met.  Optionally,  as  a  management 
action,  the  program  manager  might 
consider  a  reschedule  of  the 
program  due  to  the  slack,  which 
the  workforce  has  a  tendency  to 
expand  to  fill. 

In  the  second  condition, 
condition  yellow,  the  system 
output  is  more  extensive.  A 
critical  path  network  is 
generated  which  depicts  those 
activities  which  have  some 
probability  of  impacting  on  the 
schedule.  Additionally,  a  watch 
list  is  provided.  The  management 
action  required  in  this  condition 
is  to  insure  that  the  program 
does  not  track  along  the 
probabilistic  predictions,  since 
they  inevitably  lead  to  late 
completion.  This  is  done  by  a 
means  of.  the  watch  list.  The 


watch  list  is  simply  a  tabulation 
of  the  minimum  number  of 
activities,  beginning  with  those 
currently  underway,  that  mus’t  be 
held  to  their  deterministic  dates 
in  order  to  ensure  that  there  is 
sufficient  slack  in  the  remaining 
portions  of  the  program  to  accom¬ 
modate  the  uncertainty  without 
being  late.  This  is  essentially 
a  list  of  the  activities  that 
must  be  held  to  their  dates.  The 
list  is  then  subdivided  into  time 
periods  ano  functional  areas  so 
that  each  subordinate  is  given  a 
short  list  of  specific  items  to 
be  controlled  during  the  next 
reporting  period.  The  advantage 
of  this  approach  is  to  conserve 
management  resources  by  directing 
attention  only  to  those 
activities  with  potential  to 
cause  deviation  from  plan. 

In  the  third  condition,  condition 
red,  the  system  output  is  in  the 
form  of  a  man-machine  interface 
permitting  workaround  plan 
development  and  evaluation 
through  user  interaction  with  the 
simulation  model.  This  is 
necessary  to  accommodate  the 
management  action  necessary  to 
intervene  in  the  network  and  take 
overt  action  to  bring  the  program 
back  under  control.  In  the 
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process  of  condition  red  gaining, 
the  system  generates  a  list  of 
those  activities  that  impact  on 
the  network  in  an  adverse  way, 
the  manager  or  analyst  selects  a 
potential  corrective  action  and 
applies  it  to  that  list  of 
activities,  tests  that  action 
interactively  with  a  simulation 
and  determines  the  impact  of  his 
action.  The  system  red  module 
then  determines  if  sufficient 
action  has  been  taken  to  bring 
the'  network  back  into  a  yellow 
condition.  If  so,  this  is 
considered  an  alternative;  if 
not,  additional  impacting 
activities  are  listed  and  other 
alternative  actions  are  identi¬ 
fied,  an  action  is  selected  and 
tested,  and  the  process  is 
repeated  until  sufficient  correc¬ 
tive  action  is  taken  to  bring  the 
program  back  into  condition 
yellow.  When  this  is  done  it's 
identified  as  an  alternative  and 
documented  by  the  system.  At  the 
end  of  each  alternative  developed 
the  analyst  or  manager  is  given 
the  opportunity  to  reset  the 
scenario  and  replay  the  system 
red  analysis  to  generate  a 
different  alternative. 

After  a  number  of  these  alterna¬ 
tives  have  been  generated,  the 
program  manager  then  may  select  a 
specific  alternative.  Those 
overt  actions  selected  in  that 
alternative  are  documented  as  the 
revised  plan  in  the  system,  the 
simulation  is  run  and  a  critical 
path  and  a  watch  list  are 
generated  from  the  resultant 
condition  yellow  status.  The 
documented  alternative  then 
becomes  the  manager's  directed 
action  to  bring  the  program  back 
on  schedule. 

The  condition  red  workaround 
capability  can  also  be  applied  to 
conditions  other  than  red  and 
forms  the  basis  for  a  general 
"what  if"  gaming  capability. 


CURRENT  APPLICATIONS 

NAVSEA  has  a  number  of  program 
managers  that  currently  use  the 
Schedule  Management  System.  Most 
significant  are  the  CV  SLEP 
program  manager,  the  High  Energy 
Laser  Research  and  Development 


program  manager,  the  Mine  Counter 
Measure  Acquisition  program 
manager,  and  the  Amphibious  Ship 
Acquisition  program  manager.  SHS 
is  also  being  used  by  the  Marine 
Corp  in  the  LVT(X)  program  with 
regard  to  logistics  support. 
NAVAIR  is  currently  using  the 
cost  oriented  TRACE  concept  on  a 
number  of  programs. 


SUMMARY 

In  summary,  it  should  be  noted 
that  there  is  certainly  every 
indication  from  the  review  of  a 
large  number  of  DoD  major 
acquisitions  that  the  ability  to 
control  projects  within  cost  and 
schedule  is  a  problem.  The  most 
significant  element  in  trying  to 
control  a  schedule  problem  is  to 
identify  and  understand  it, 
followed  by  timely  determination 
of  exactly  where  and  how  much 
specific  intervention  is  required 
to  bring  the  program  under 
control. 

There  is  a  Navy  policy  which 
directs  managers'  attention  to 
evaluation  and  control  of  risk 
but  there  is  no  official  position 
on  how  this  is  to  be  done. 
Currently,  the  closest  approxima¬ 
tion  to  actual  control  is  a 
requirement  that  subjective 
reporting  be  done  in  accordance 
with  the  NS ATS  program  manager 
appraisal  requirements.  However, 
the  Navy  Acquisition  Research 
Council  is  sponsoring,  through 
the  Defense  System  Management 
College,  the  development  of  a 
Risk  Assessment  Management 
Handbook. 

The  Naval  Sea  Systems  Command  has 
over  the  years  been  involved  in 
*:he  development  of  risk  analysis 
tools  and  has  been  involved  in 
the  application  of  these  tools  to 
major  programs.  SMS  is  one  of 
these  tools  that  is  successfully 
used  on  a  number  of  NAVSEA 
programs.  It  permits  the  manager 
to  incorporate  his  plan  in  a 
network  format,  and,  by  applica¬ 
tion  of  progress  vs.  plan 
reporting  by  qualified,  exper¬ 
ienced  people  the  manager  is  able 
to  accurately  assess  program 
status.  One  of  the  most 
important  elements  of  any  quanti- 
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tative  analysis  scheme  is  the 
grass  roots  technical  knowledge 
of  the  program  in  order  to  assign 
deterministic  schedules  and  to 
assign  probabilistic  range  values 
for  any  activity.  The  Monte 
Carlo  simulation  identifies  those 
areas  of  uncertainty  which  have 
the  potential  to  impact  program 
objectives/  quantifies  this 
impact,  and  directs  management 
attention  to  those  minimum  areas 
that  must  be  controlled  in  order 
to  maintain  schedule  adherence. 
In  the  situation  where  activity 
progress  is  no  longer  contained 
within  the  plan,  the  system 
indicates  whether  that  lack  of 
containment  will  directly  contri¬ 
bute  to  an  overall  project 
overrun  of  schedule.  In  those 
cases  where  objectives  are  not 
effected  management  action  is  not 
required,  in  all  other  cases 
management  action  is  required. 
Where  action  is  required  the 
system  permits  the  manager  to 
experiment  and  simulate  various 
management  actions  thus 
,*  developing  alternative  solutions 
to  the  problem  considering 
elements  of  cost,  time  and 
feasibility.  The  program  manager 
can  select  the  best  alternative 
and  insert  the  appropriate 
changes  into  the  model.  The 
system  then  documents  new  dates 
that  become  his  new  schedule 
direction  to  the  program  staff. 

In  conclusion,  it  can  be  stated 
that  although  quantitative  risk 
analysis  is  being  used  by  some 
program  managers,  it  is  apparent 
th-at  many  program  managers 
neither  support  nor  appreciate 
the  total  benefits  which  would 
accrue  with  the  application  of 
quantitative  risk  management  to 
their  program.  If  quantitative 
risk  analysis  is  to  be  used  cost 
effectively  as  a  management  tool, 
it  requires  high  level  technical 
and  management  support  and 
involvement.  There  can  be  little 
doubt  that  given  the  Carlucci 
initiatives  and  congressional 
sensitivity  to  cost  overruns 
associated  with  major  acquisition 
programs,  that  the  trend  for 
acquisition  program  managers  is 
toward  tha  utilization  of  quanti- 
tative  risk  management. 
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ABSTRACT 

TRACE-P,  Total  Risk  Assessing  Cost  Estimate 
for  Production,  is  intended  as  a  contingency¬ 
funding  vehicle  for  the  first  three  years  of 
production  of  designated  systems.  TRACE-P 
extends  the  TRACE  concept  of  RDTE  to  Produc¬ 
tion,  and  serves  to  quantify  risks  in  terms  of 
their  cost  impact  on  designated  systems.  A 
mechanism  is  proposed  here  for  generating  such 
risk  costs.  It  extends  the  use  and  applica¬ 
tion  of  the  contractor's  Work  Breakdown  Struc¬ 
ture  (WBS)  in  identifying  risk  prone  areas, 
and  combines  the  WBS  with  probabilistic  net¬ 
working  techniques  to  create  a  data  structure 
which  generates  risk  costs  for  the  designated 
program^ 


PURPOSE 

The  purpose  of  this  paper  is  to  show  how  the 
Venture  Evaluation  and  Review  Technique 
networking  model  can  be  combined  with  the  con¬ 
tractor  Work  Breakdown  Structure  in  a  way  that 
provides  the  Program  Manager  with  a  powerful 
tool  in  determining  potential  risk  costs  to 
his  program. 


INTRODUCTION 


TRACE-P,  Total  Risk  Assessing  Cost  Estimate 
for  Production,  is  intended  as  a  contingency¬ 
funding  vehicle  for  the  first  three  years  of 
production  of  designated  systems.  TRACE-P 
continues  the  TRACE  concept  of  RDTE,  and  serves 
to  quantify  risks  in  terms  of  their  cost  impact 
on  designated  systems.  The  proposed  mechanism 
for  generating  such  risk  costs  involves  no  new 
reporting  format.  In  fact,  it  merely  extends 
the  use  and  application  of  an  existing  report¬ 
ing  vehicle,  the  contractor's  Work  Breakdown 
Structure  (WBS).  The  reasons  for  using  the 
WBS  are  plain: 

1.  Virtually  all  risk-prone  activities  are 
performed  by  the  contractor,  not  Government. 
Government  is  responsible  for  managing  programs 
with  risk;  contractors  encounter  risk  in  actual 
execution  of  these  programs. 

2.  The  WBS  hierarchy  is  a  very  convenient 
format  to  use  in  identifying  those  contractor 
activities  whfth  are  more  risk-prone  than 


others.  Thus,  use  of  documents  such  as  the 
contractor's  Production  Plan,  Development  Test/ 
Operational  Test  reports.  Production  Readiness 
Reviews,  Technical  Data  Packages,  etc.  in  con¬ 
junction  with  the  WBS  will  allow  simple  identi¬ 
fication  of  risk  prone  areas. 

The  WBS  is  seen  to  be  a  powerful  means  of  iso¬ 
lating  risk  to  those  pertinent  work  areas. 
However,  the  WBS  in  and  of  itself  provides  an 
incomplete  picture  of  any  contractual  effort. 
This  is  because  although  the  WBS  shows  the 
hierarchial  organization  of  tasks,  it  does  not 
show  the  manner  in  which  these  tasks  are  exe¬ 
cuted,  nor  the  relation  of  the  tasks  to  each 
other  from  a  schedule  network  perspective.  The 
full  potential  of  the  WBS  as  an  analytical 
tool  is  therefore  limited  ’f  we  restrict  our 
use  of  it  to  its  hierarchial  form. 


POLICY  CONSIDERATIONS  FOR  THE  WBS 

At  present,  the  contractor's  WBS  is  used  by  the 
government  as  the  basis  for  tracking  contractor 
cost  and  performance.  In  fact,  Cost/Schedule 
Control  Systems  Criteria  (C/SCSC)  reporting  is 
essentially  one  of  the  few  uses  to  which  the 
CWBS  is  applied  in  project  management.  In  many 
instances,  an  examination  of  contractor  sche¬ 
dules  show  that  activities  and  milestones  often 
relate  to  Contractor  Data  Requirements  List 
items  more  than  they  do  to  the  WBS.  This  lack 
of  correlation  can  lead  to  needless  confusion. 
Further,  because  schedule  information  Hoes  not 
track  with  the  WBS,  any  projections  addressing 
schedule  or  cost  uncertainty  will  of  necessity 
come  from  two  diverse  sources  -  the  contract¬ 
or's  schedule  and  the  WBS  work  packages,  re¬ 
spectively.  What  is  needed  is  a  means  of  tying 
cost  and  schedule  considerations  together,  and 
this  objective  can  be  readily  obtained  by  a 
change  in  the  use  of  the  contractor's  WBS  and 
schedule  network  data. 

It  is  proposed  that  contractors  be  required  to 
submit  schedule  network  diagrams  of  their 
activities  and  milestones  so  that  the  follow¬ 
ing  minimum  criteria  are  met: 

(1)  Each  WBS  element  corresponds  to  one 
arc  at  an  appropriate  level.  The  coarsest 
level  of  detail  should  be  level  3,  and  where 
specified  by  the  government,  should  be  4  or 
lower  if  finer  detail  is  required. 
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(2)  For  each  arc,  the  contractor  must  pro¬ 
vide  its  expected  cost  and  duration,  with  all 
costs  expressed  in  common  units  and  all  times 
expressed  in  common  units.  The  expected  cost 
should  be  readily  available  because  in  most 
cases  that  will  correspond  to  the  value  of  some 
work  package. 

(3)  The  network  should  be  structured  so 
that  time  phasing  of  activities  and  milestones 
will  be  readily  apparent;  the  interrelation¬ 
ships  between  activities  and  milestones  (i.e., 
network  logic)  also  should  be  readily  apparent. 

(4)  Activities  which  cannot  be  included  as 
part  of  the  WBS  but  which  do  affect  time  and 
cost  must  be  included  in  the  network. 

(5)  The  network  must  span  a  period  of  time 
covering  contract  award  to  last  delivery,  and 
the  sum  of  the  costs  for  all  arcs  in  the  net¬ 
work  must  equal  the  contract  cost  or  appropri¬ 
ate  financial  measure. 

A  few  comments  are  in  order  here.  First,  the 
use  of  the  WBS  as  the  basis  for  a  schedule  net¬ 
work  to  be  submitted  by  the  contractor  is  cer¬ 
tainly  achievable.  The  imposition  of  such  a 
requirement  on  him  should  not  be  any  great 
hardship,  because  such  information  must  already 
be  at  hand.  For  example,  the  contractor  must 
know  how  his  work  is  organized  and  he  must  have 
a  fairly  good  idea  of  how  much  time  and  money 
each  piece  of  the  work  will  require.  From  the 
government  perspective  this  is  a  very  reason¬ 
able  expectation.  However,  we  must  next  con¬ 
sider  the  contractor's  concerns.  Often  the  WBS 
is  devised  in  such  a  way  that  it  simply  does 
not  make  sense  to  use  the  WBS  for  presenting 
schedule  data.  Consequently,  the  contractor  is 
forced  to  present  schedule  data  in  a  manner 
different  from  the  WBS.  If  the  method  proposed 
here  is  to  work,  government  managers  must 
choose  and  devise  WBS  elements  in  such  a  way 
that  their  portrayal  in  schedule  format  becomes 
feasible.  One  way  this  can  be  accomplished  is 
if  the  WBS  is  not  strictly  bound  to  the  hard¬ 
ware/software  configuration  of  a  system  and  its 
corresponding  subsystems.  If  instead  the  WBS 
is  portrayed  to  have  as  its  subelements  the 
activities  associated  with  any  particular  sub¬ 
system,  then  it  will  be  a  simple  matter  for  the 
contractor  to  provide  the  WBS-derived  schedule. 
For  example,  if  a  piece  of  electronics  equip¬ 
ment  were  to  be  developed,  the  WBS  for  that 
equipment  might  include  subelements  headed 
"Design",  "Breadboard",  "Test".  In  this  way 
all  WBS  elements  will  be  included  in  the  sche¬ 
dule  network.  It  is  again  stressed  that  if 
this  concept  is  to  work,  government  and  con¬ 
tractual  management  alike  are  going  to  have  to 
view  the  WBS  as  a  vehicle  for  other  than  CSSRs, 
CPRs  and  the  like.  The  WBS  concept  in  the 
presently  proposed  form  is  expanded  to  provide 
total  contract  and,  therefore,  total  project 
representation  in  a  manner  whi*ch  unifies  cost 


and  schedule  considerations.  That  is  why  the 
policy  regarding  the  use  of  the  WBS  as  well  as 
the  means  of  reporting -schedule  and  cost  needs 
to  be  reviewed  and  changed.  Only  then  will  the 
benefits  that  the  remainder  of  this  report  dis¬ 
cusses  be  realized.  Extending  the  use  of  the 
WBS  will  as  a  minimum  provide  the  government 
with  a  data  base  for  its  TRACE-P  analyses. 


THE  WBS  DATA  BASE  AND  VERT 

Many  current  tools  of  generating  risk  costs 
involve  analysis  of  the  WBS  in  its  tabular 
form,  or  at  best  a  bar  chart  schedule  which 
lists  each  activity  in  a  more  or  less  stand¬ 
alone  fashion.  The  Venture  Evaluation  Review 
Technique  (VERT)  eliminates  these  deficiencies 
by  allowing  program  activities  to  be  linked 
together  in  a  symbolic  network  which  is  then 
probabilistically  exercised  for  many  (several 
hundred)  iterations,  dynamically  testing  pro¬ 
gram  activities  and  their  interfaces.  Unlike 
other  networking  techniques  which  have  fixed 
input  data,  VERT  allows  for  functional  rela¬ 
tionships  to  be  defined,  i.e.,  the  cost  of  one 
activity  may  be  a  function  of  the  time-or  man¬ 
power-loading  of  that  activity,  or  of  other 
related  activities.  This  allows  a  more  reali¬ 
stic  modelling  to  be  conducted  of  the  contac¬ 
tor's  work,  thereby  providing  a  refined  measure 
of  the  associated  risk  costs  compared  to  other 
*  analytical  tools.  The  only  additional  require¬ 
ment  that  use  of  the  VERT  techniques  would  im¬ 
pose  on  contractor  personnel  is  that  they 
provide  WBS  schedule  data  in  network  form, 
similar  to  PERT-type  diagrams.  We  repeat  our 
assertion  that  these  data  should  be  readily 
available  from  the  contractor  because  the 
various  cost  account  managers  have  to  know  how 
they  are  spending  money  on  the  work  being  per¬ 
formed.  Once  the  data  are  provided  to  the 
Government  in  this  format,  the  Government  ana¬ 
lyst  can  structure  the  VERT  network  and  conduct 
the  necessary  activities  needed  in  preparing 
the  numerical  data  to  be  exercised  by  the  net¬ 
work  logic.  VERT  would  then  generate  histogram 
data  on  cost  and  time  which  would  predict  the 
contractor's  performance  based  on  the  input 
data. 


SAMPLE  CASE  TRACE-P  USING  VERT 

To  illustrate  the  application  of  these  proce¬ 
dures  to  generating  TRACE-P  estimates,  we  con¬ 
sider  the  hypothetical  System  X  whose  WBS  and 
production  schedule  are  shown  in  Figure  1.  Sy- 
tem  X  has  four  major  subsystems  which  are  pro¬ 
duced  in  parallel,  and  then  integrated  and 
tested  before  delivered  to  the  government.  In 
the  past,  a  TRACE-P  estimate  for  System  X  would 
have  been  generated  by  having  personnel  with 
appropriate  expertise  examine  each  WBS  element 
or  else  each  risk  element,  and  quantify  the 
risk  for  each  element  in  the  form  of  a  numeri- 
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cal  multiplier.  For  example,  if  element  XA 
costs  $1  and  it  is  determined  that  a  risk  mul¬ 
tiplier  of  1.25  is  appropriate  for  XA,  then 
XA's  contribution  to  the  TRACE-P  is  $1.25. 

The  full  TRACE-P  would  be  the  sum  of  all  such 
products.  In  this  manner,  each  element  con¬ 
tributes  its  portion  to  the  TRACE-P  in  the 
form  of  a  point  estimate;  the  TRACE-P  for  Sys¬ 
tem  X  is  also  a  point  estimate  which  is  the 
sum  of  the  point  estimates  for  each  WBS  ele¬ 
ment.  So  although  the  risk  factor  method  is 
useful  in  identifying  risk  areas  and  their 
contributions  to  TRACE-P,  nevertheless  the 
outcome  of  this  type  of  approach  is  one  number, 
a  point  estimate. 

In  the  method  proposed  here,  the  contractor 


would  provide  the  Government  schedule  informa¬ 
tion  on  the  WBS  in  the  network  form  of  Figure 
2.  Each  arc  in  the  network  corresponds  to  an 
element  of  the  WBS,  and  therefore  the  associa¬ 
ted  cost  with  that  element  can  be  readily  pro¬ 
vided  by  the  contractor.  Uncertainties  in 
cost  and  schedule  can  now  be  examined  in  the 
light  of  this  network  representation;  for 
example,  in  the  network  for  Systems  X's  WBS, 
integration  and  test  cannot  begin  until  after 
unit  #1  for  each  subsystem  has  been  fabricated; 
this  in  turn  affects  the  start  of  production 
deliveries.  If  there  is  a  stretchout  in  the 
production  schedule,  the  cost  associated  with 
that  affected  portion  of  the  schedule  can  be 
modelled  in  VERT  as  a  function  of  time,  and 
the  spread  in  time  values  will  provide  a  more 


46 


System  I  VB3  Network 


Figure  2. 


deterministic  basis  for  the  TRACE-P  generated. 
The  spreads  in  time  and  cost  can  be  determined 
by  consultation  with  Government  technical  and 
contractual  experts  familiar  with  the  WBS 
element  under  scrutiny.  The  analyst  can  in¬ 
corporate  this  information  in  the  VERT  data 
base,  and  then  by  exercising  the  WBS  network 
with  VERT,  a  measure  of  System  X's  TRACE-P 
costs  can  be  obtained. 

Table  1  is  a  list  of  hypothetical  input  values 
for  time  and  cost  that  VERT  would  use  in  simu¬ 
lating  System  X's  project.  The  time  parameters 
are  given  in  a  form  suited  to  the  use  of  the 
triangular  probability  density  distribution; 
however,  VERT  permits  the  use  of  many  distribu¬ 
tions,  and  if  another  distribution  were  more 
applicable  for  modelling  time,  such  as  the 


exponential,,  normal  or  binomial,  the  data  ap¬ 
plicable  to  their  use  could  be  easily  formatted 
for  execution  by  VERT.  Regardless  of  the 
distribution  used,  the  data  would  have  been  ob¬ 
tained  from  detailed  conversations  with  area 
specialists  so  as  to  assure  inclusion  of  their 
expert  opinions  in  forming  the  VERT  data  base. 
The  same  would  also  apply  to  the  costs  being 
modelled. 

Cost  distributions  are  the  central  issue  when 
it  comes  to  discussing  TRACE-P,  and  for  the  WBS 
elements  of  System  X,  it  was  decided  to  choose 
their  representative  costs  as  being  linear 
functions  of  the  time  required  to  complete 
each  activity,  thereby  illustrating  the  great 
flexibility  of  the  VERT  system  in  generating 
TRACE-P  figures.  To  further  clarify,  consider 
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Input  Data 
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Table  1. 


the  cost  expression  for  the  first  unit  produc¬ 
tion  of  subsystem  A.  The  expression  is: 

C  =  100  +  200  (TxA  -1). 

The  time  data  for  subsystem  A  indicates  a  most 
likely  and  also  a  minimum  requirement  of  1 
month  to  produce  the  first  unit  before  it  is 
sent  forward  for  integration  and  test.  The 
cost  relationship  here  is  structured  In  such 
a  way  that  if  the  time  required  to  produce 
subsystem  A's  first  unit  exceeds  1  month,  a 
penalty  of  J200K  times  the  excess  measured 
in  months  exceeding  1  month  will  be  incurred. 
This  would  correspond  to  a  real  world  situa¬ 
tion  where  the  contracts*  would  need  to  hire 
many  highly  skilled  workers,  or  make  addition¬ 
al  capital  investments  to  assure  minimal  sch- 


dule  slippage.  However,  if  there  is  no  slip¬ 
page,  no  cost  penalty  is  incurred.  Each  sub¬ 
system  has  its  own  cost  penalty.  As  VERT  ex¬ 
ercises  the  System  X  WBS  network,  random  time 
values  for  the  respective  WBS  subelements  are 
incurred  for  each  iteration  of  VERT,  thereby 
generating  different  cost  penalties. 

If  the  contractor  were  100%  certain  of  meeting 
his  schedule,  there  would  be  no  variability 
in  time  and  hence  no  cost  penalties.  The  total 
contractual  cost  would  be  $4100K,  which  is  the 
sum  of  the  constant  parts  of  all  the  WBS  sub- 
element,  costs  in  Table  1.  However,  there 
schedule  uncertainty,  which  is  reflected  in  the 
fact  that  the  time  data  for  each  subelement  of 
System  X  is  described  by  a  probability  distri¬ 
bution.  This  In  turn  causes  various  cost  pen¬ 
alties  to  be  incurred  for  each  iteration  of  the 
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System  X  network  by  VERT.  After  the  number  of 
iterations  is  completed,  VERT  will  generate 
histograms  of  cost  data  -  by  sequential  time 
period  and  for  the  program's  full  duration  - 
which  the  Program  manager/analyst  may  use  in 
selecting  an  appropriate  risk  level  for  TRACE- 
P  funding.  Figures  3-6  are  histograms  genera¬ 
ted  by  VERT  for  months  0-12,  12-24,  24-36,  and 
0-36  of  the  program,  providing  the  PM  with  an¬ 
ticipated  yearly  costs  as  well  as  anticipated 
program  costs.  If,  for  example,  the  Program 
Manager  of  System  X  wishes  to  be  conservative 
during  the  first  year  of  the  program,  he  might 
pick  the  90%  point  of  the  histogram  for  months 
0-12.  This  can  be  found  by  interpolating  the 
cumulative  distribution  function  values,  which 
bracket  the  90%  point,  and  comes  out  to  $4404K 
for  year  1.  The  meaning  of  this  choice  is 
simply,  that  of  all  the  cost  values  generated 
by  VERT  for  year  1  of  System  X,  the  value 
54404K  was  exceeded  only  for  10%  of  those  iter¬ 
ations,  and  therefore,  it  exceeded  90%  of  the 
cost  values. generated  for  that  year.  By  se¬ 
lecting  a  large  number  of  iterations  we  can  be 
statistically  confident  that  costs  will  fall 
within  this  arena,  providing  we  have  an  accur¬ 
ate  representation  of  subelement  costs  and 
schedules.  Use  of  the  W8S  helps  to  assure  this 
aspect  of  getting  an  accurate  handle  on  TRACE- 


T: 
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Analagous  choices  of  percentile  points  can  be 
made  for  years  2  and  3  of  System  X.  In  this 
manner,  the  risk  funding  level  may  be  lowered 
for  years  2  and  3  If  the  PM  feels  such  actions 
are  warranted.  The  overall  program  risk  fund¬ 
ing  level  may  be  found  by  summing  costs  for 
years  1,  2  and  3  and  reading  the  value  obtain¬ 
ed  off  the  overall  program  cost  histogram, 
Figure  6.  To  again  illustrate,  the  90%  point 
for  year  1  was  found  to  be  $4404K.  For  year  2 
(months  12-24),  let  us  read  directly  off  the 
histogram.  The  81.8%  point  is  $941K,  and  let 
us  suppose  the  PM  Is  satisfied  with  this  fig¬ 
ure,  i.e.,  of  the  cost  generated  by  VERT  for 
year  2,  they  did  not  exceed  $941K  for  81.8%  of 
the  total  iterations.  For  year  3,  suppose  the 
PM  selected  the  73.3%  point  which  reads  as 
$56K.  The  sum  of  these  3 "figures,  4404  +  941  + 
66,  is  $ 5401K ,  and  this  corresponds  to  an  over¬ 
all  program  risk  funding  level  of  about  90%,  as 
shown  in  Figure  6.  That  is  to  say,  in  order  to 
be  90%  confident  that  System  X's  contractor 
costs  will  not  exceed  his  budget,  the  PM  would 
need  to  have  on  hand  $5401K,  or  31.7%  above 
the  initial  projected  cost  of  $4100K.  Whether 
or  not  such  a  contingency  funding  level  is 
appropriate  for  a  system  entering  production 
will  not  be  discussed  here.  The  example  chosen 
had  purposely  built-in  severe  cost  penalties 
to  illustrate  the  nature  of  TRACE-P  issues.  If 
tne  PM  wished  to  be  less  conservative  in  this 
example,  he  might  be  willing  to  go  for  a  70% 
confidence  level.  The  new  total  cost  as  read 
from  Figure  6  would  then  be  55190k,  or  26.6% 
above  the  contract  budget.  Tne  TRACE-P  defer¬ 


ral  for  the  3  years  would  be  $109QK  ($5190K  - 
4100K).  The  PM  could  then  allocate  the  TRACE- 
P  deferral  among  each  program  year,  verifying 
that  when  the  deferral  is  added  to  the  base¬ 
line  for  all  three  years,  tney  sum  up  to  the 
$5190K.  Whatever  course  is  taken,  tne  WBS 
network  approacn  permits  the  PM  to  make  diffi¬ 
cult  decisions  with  more  useful  information  at 
his  disposal.  With  cost  becoming  an  increas¬ 
ingly  scrutinized  arena,  the  VERT-WBS  method¬ 
ology  for  generating  TRACE-P  estimates  cannot 
be  ignored. 
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INTRODUCTION 

/ 

The  problem  of  dealing  with  cost  growths 
in  its  acquisition  of  weapon  systems  has 
haunted  the  Defense  Department  for  years  and 
probably  will  continue  to  do  so  until  either 
irastic  changes*  are  made  in  the  acquisition 
process  or  until  we  just  stop  keeping  track 
of  costs.  Since  neither  of  these  is  likely 
to  occur,  the  problem  of  defense  systems 
acquisition  costs  exceeding  original  esti¬ 
mates  and  budgets  is  probably  here  to  stay. 
Nevertheless,  there  are  a  number  of  actions 
that  can  be  taken  to  improve  the  process  and 
to  reduce  the  magnitude  of  future  cost 
growths. 

This  paper  deals  with  just  one  of  these 
actions  as  it  describes  a  research  project 
conducted  by  the  Defense  Systems  Management 
College'ToSMCT to  develop  a  handbook  designed 
to  assist  acquisition  management  personnel 
in  selecting  and  Implementing  risk  assess^ 
ments.  The  purpose  of  this  paper  is:  1 )  to 
briefly  describe  tha£_research  project^^a^ 

Its  primary  product,-  the  handbookjand  2)  to 
share  some  of  the  findings  from  this  research 
that,  while  inappropriate  for  Inclusion  in 
the  handbook,  should  nevertheless,  be  of 
interest  to  acquisition  management  personnel. 
The  objective  of  sharing  this  other  infor¬ 
mation  is  to  suggest  areas  whereby  risk 
assessments  can  become  a  more  effective  tool 
in  fighting  the  problem  of  cost  growth. 


BACKGROUND 

One  of  the  primary  thrusts  of  the 
Department  of  Defense  Acquisition  Improve¬ 
ment  Program  (AIR)  was  to  develop  specific 
actions  whose  implementation  would  reduce 
the  number  of  weapon  system  programs  oxpe- 


•Examples  of  the  magnitude  of  the  changes 
we  are  referring  to  are:  1)  the  creation 
of  a  permanent,  professional,  civilian 
organization  to  manage  the  acquisition  of 
weapons  systems  for  all  the  services  and 
2)  the  extension  of  the  budget  cycle 
from  one  year  to  two. 


riencing  cost  growths.  One  such  action  was 
AIP  Action  Number  11.  This  action  addressed 
the  problem  of  program  budgets  failing  to 
adequately  account  for  the  technological 
risks  inherent  in  the  development  and  pro¬ 
duction  of  today's  complex  weapon  systems. 
This  action,  commonly  referred  to  as  "Budget¬ 
ing  for  Technological  Risk,"  recommended 
that  efforts  within  the  Department  of 
Defense  to  quantify  risk  be  increased  and 
that  the  use  of  budgeted  funds  to  deal  with 
uncertainty  be  expanded.  Specifically,  this 
action  directed  that  the  Services  budget 
funds  for  risk  and  that  "each  Service  should 
review  the  TRACE  (Total  Risk  Assessing  Cost 
Estimate)  concept  and  either  adopt  it  0" 
propose  an  alternative  for  their  use." 

(1  p.  12) 

Today,  the  issue  of  budgeting  for  risk 
is  continuing  to  receive  emphasis  at  the 
highest  levels  in  DoD.  Deputy  Secretary  of 
Defense  Thayer  has  Included  "Budgeting  for 
Technological  Risk"  in  the  list  of  actions 
he  has  recently  consolidated  into  six  areas 
for  increased  management  attention.  (2) 

The  actions  were  selected  for  inclusion  in 
this  consolidated  list  because  of  their 
importance  to  the  overall  defense  mission. 

The  Services,  along  with  the  designated 
"lead  office"  within  the  Office  of  the 
Secretary  of  Defense,  will  develop  more 
detailed  objectives  and  actions  for  each 
consolidated  area.  The  designated  "lead 
office"  for  "Realistic  Budgeting"  (the 
consolidated  area  now  containing  AIP  Action 
Number  11)  is  the  Assistant  Secretary  of 
Defense,  Comptroller.  Progress  in  imple¬ 
menting  the  more  detailed  actions  will  be 
reviewed  on  a  regular  basis  by  Deputy 
Secretary  Thayer  during  meetings  of  the  DoD 
Council  on  Integrity  and  Management  Improve¬ 
ment. 

In  an  attempt  to  assist  the  implemen¬ 
tation  of  the  "Budgeting  for  Technological 
Risk"  action,  in  the  fall  of  1982  DSMC 
embarked  on  a  contracted  effort  to  develop  a 
Handbook  of  Risk  Assessment/Budgeting 
Techniques.  The  objectives  of  this  project 
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were  to  identify  the  risk  assessment  tech¬ 
niques  actually  being  used  by  defense 
acquisition  management  activities  and  to 
describe  these  techniques  along  with  their 
advantages  and  disadvantages  in  a  handbook. 

The  handbook  was  also  to  include  information 
to  help  the  user  select  and  implement  a 
technique  chosen  for  his  particular  appli¬ 
cation.  This,  then,  establishes  the  back¬ 
ground  and  rationale  for  the  DSMC  research 
effort. 

RESEARCH  APPROACH 

In  the  next  few  paragraphs,  the  approach 
taken  in  this  research  project  will  be 
described  to  form  a  basis  for  understanding 
the  findings  and  recommendations  contained 
later  in  this  paper. 

Under  the  DSMC-managed  research  contract, 
the  first  step  taken  was  to  conduct  an 
extensive  search  of  the  existing  literature 
in  the  areas  of  risk  assessment  and  budgeting 
for  risk.  The  purpose  of  this  search  was 
to  identify  the  available  techniques, 
discover  where  and  how  they  were  being 
employed  and  commence  the  development  of  a 
bibliography  to  assist  the  ultimate  users  of 
the  handbook  In  pursuing  areas  of  particular 
interest. 

After  review  of  the  data  collected 
through  the  literature  search,  It  was 
determined  that  a  survey  of  acquisition 
management  activities  was  necessary  to  get  a 
better  indication  of  on-going  risk  assess¬ 
ment  activities.  A  plan  listing  the  Program 
Management  Offices  (PMOs)  to  be  surveyed 
was  developed  using  the  following  criteria  - 
Service,  type  of  PMO  (l.e.,  size,  degree  of 
matrix  support,  etc.),  type  of  weapon  system 
and  status  of  the  program.  This  led  to  a 
plan  to  research  over  50  PMOs  at  some  15 
different  defense  activities.  The  plan  also 
Included  interviews  of  the  appropriate 
functional  support  offices,  In  addition  to 
the  PMOs,  at  the  various  activities  visited. 

Preliminary  evaluation  criteria  were 
developed  for  the  purpose  of  determining  the 
"usefulness"  of  the  risk  assessment  tech¬ 
niques  in  use.  These  criteria  formed  the 
basis  of  the  questions  asked  during  the 
survey.  The  Initial  set  of  criteria  In¬ 
cluded  -  soundness  of  theory,  availability 
of  necessary  data,  resources  required, 
availability  of  software,  speed  of  the 
solution,  perceived  value  to  the  decision¬ 
making  process  and  credibility  of  the 
technique. 

After  completing  the  survey,  the 
collected  data  were  analyzed  to  categorize 
the  techniques  discovered  and  to  describe 


their  advantages  and  disadvantages  as  identi¬ 
fied  by  the  personnel  interviewed.  A 
refined  set  of  evaluation  criteria  was  used. 
Some  of  the  preliminary  criteria  were 
dropped  and  some  new  ones  were  added 
resulting  in  the  following  list  -  soundness 
of  theory,  resources  required  and  value  as  a 
decision  aid  (a  subjective  evaluation  of 
several  factors  by  PMO  personnel). 

A  draft  version  of  the  handbook  was 
developed  based  on  the  results  of  the  data 
analysis  and  technique  evaluation  efforts. 
This  handbook  draft  was  reviewed  in  detail 
by  individuals  representing  a  broad  cross- 
section  of  the  acquisition  management 
community  and  modified  accordingly.  The 
revised  handbook  has  been  published  by  DMSC 
and  distributed  to  various  acquisition 
management  offices,  including  of  course, 
those  who  assisted  in  the  survey.  Incor¬ 
poration  of  the  handbook  into  the  Defense 
Technical  Information  Center  (DTIC)  is  in 
process . 

HANDBOOK  DESCRIPTION 

In  the  following  paragraphs,  a  des¬ 
cription  of  the  contents  of  the  handbook 
will  be  given.  The  purpose  of  this  des¬ 
cription  is  to  give  the  reader  of  this  paper 
a  general  understanding  of  the  type  of 
material  contained  in  the  handbook  and  its 
organization,  but  not  to  provide  a  compre¬ 
hensive  summary  of  its  contents. 

The  handbook  was  prepared  with  basic¬ 
ally  two  major  sub-divisions.  The  first 
section  of  six  chapters  was  written  to  give 
the  acquisition  manager  a  concise,  easy-to- 
read,  management-level  overview  of  risk 
assessment  and  what  it  should  mean  to  him.* 
The  purpose  of  this  first  section  was  to 
interest  the  manager  in  applying  risk 
assessment  to  his  program  and  to  give  him 
the  background  to  do  so  through  the  use  of 
the  second  section  of  the  handbook  -  the 
Appendices.  The  Appendices  were  developed 
to  provide  the  interested  user  with  more 
details  in  selected  areas  to  aid  implemen¬ 
tation  of  a  risk  assessment  technique. 

The  entire  handbook,  and  particularly 
the  first  six  chapters,  was  written  as¬ 
suming  the  prime  user  would  be  a  manager 
within  the  PMO  who  either  had  been  charged 


*  Whenever  in  this  paper  "man,"  or  "men," 
or  their  related  pronouns  appear,  either 
as  words  or  parts  of  words,  they  have 
been  used  for  literary  purposes  and  are 
meant  in  their  generic  sense. 


with  the  responsibility  of  assessing  his 
program's  risk  or  who  felt  that  his  program 
(or  project)  could  benefit  from  a 
risk  assessment.  It  was  assumed  that  the 
targeted  user  would  be  experienced  in 
acquisition  management,  but  would  be 
relatively  unfamiliar,  or  at  least  not 
current,  in  the  details  of  conducting  risk 
assessments  such  as  the  modeling  or  inter¬ 
viewing  techniques  required.  Hopefully, 
the  experienced  analyst  will  also  find  some 
of  the  Appendices  of  interest.  It  was  not 
the  intent  of  the  handbook,  however,  to 
provide  stand-alone,  detailed  instructions 
on  "how  to  implement"  any  one  risk  assess¬ 
ment  technique.  The  Bibliography  provides 
references  for  details  associated  with  all 
of  the  techniques  discussed. 

The  first  six  chapters  lead  the  user 
through  1)  an  Introduction  -  a  description 
of  the  handbook's  purpose  and  scope,  2)  a 
discussion  of  what  risk  assessment  means. 
Including  a  very  limited  review  of  some 
elementary  concepts  in  probability,  3)  a 
general  analysis  of  the  costs  and  benefits 
of  conducting  risk  assessments  and  a  dis¬ 
cussion  of  why  a  manager  should  consider 
conducting  one,  4)  a  presentation  and  brief 
description  of  the  techniques  available,  5) 
Information  to  assist  in  the  selection  of  a 
particular  technique  and,  finally  6)  a 
discussion  of  wherp  ;  manager  could  f4,id 
support  and/or  assistance  in  implementing  a 
technique  or,  his  program.  The  second 
section  rf  the  handbook  is  made  up  of  a 
number  of  Appendices  that  provide  ampli¬ 
fying  details  in  several  areas.  In 
addition  to  the  Bibliography,  Definition  of 
Terms  and  List  of  Acronyms,  there  are 
Appendices  that:  1)  address  the  problem  of 
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quantifying  the  opinions  of  experts  and 
using  this  data  in  risk  assessments,  2) 
provide  a  further  level  of  detailed  des¬ 
cription  of  the  available  techniques,  3) 
describe  the  DoD  and  Service  policy  state¬ 
ments  associated  with  risk  assessment  and 
4)  discuss  the  budget  policy  concept  that 
specifically  incorporates  risk  assessment, 
i.e.,  TRACE. 

RESEARCH  FINDINGS 

The  findings  reported  in  this  section 
result  from  the  analysis  of  the  data  col¬ 
lected  during  the  survey  of  Army,  Navy  and 
Air  Force  system  acquisition  management 
offices  in  December,  1982  and  January, 

1983.  Interviews  were  successfully  com¬ 
pleted  in  a  total  of  46  PMOs  and  9  func¬ 
tional  support  offices  using  a  questionnaire 
briefly  described  in  the  RESEARCH  APPROACH 
section  of  this  paper.  The  findings  have 
been  categorized  and  will  be  discussed  in 
the  following  areas:  the  extent  of  use  of 
risk  assessment,  the  techniques  being  used 
and  their  reported  advantages  and  dis¬ 
advantages. 

A.  Extent  of  Use  of  Formal , 

Quantitative  Risk  Assessments 

3efore  describing  where  formal,  quan¬ 
titative  risk  assessments  were  found  in 
use,  a  working  definition  for  these  terms 
must  be  established.  As  used  in  this 
research  effort,  in  the  handbook  and  in 
this  paper,  risk  analysis  can  be  thought  to 
consist  of  the  three  activities  -  risk 
assessment,  risk  reduction  and  risk  manage¬ 
ment  as  shown  in  Figure  1  below  taken  from 
(6).  Risk  assessment  then,  consists  of 
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evaluating  an  existing  plan  and  any  alter¬ 
native  plans  that  may  be  subsequently 
developed.  A  formal  assessment  is  considered 
to  be  one  that  utilizes  the  classical 
system  analysis  approach  by  attacking 
complex  problems  through  analyzing  the 
individual  elements  that  make  up  the  problem, 
determining  the  relationships  among  these 
elements  and  then  formulating  a  model  of 
the  problem  to  study.  A  quantitative 
assessment  attempts  to  develoo  mathematical 
descriptions  of  the  relationships  that 
exist  among.the  problem's  elements;  i.e., 
to  formulate  a  model  in  mathematical  terms. 

Where  then,  were  formal,  quantitative 
risk  assessments  found  to  be  in  use?  Of 
the  46  PMOs  surveyed  in  the  Army,  Navy  and 
Air  Force,  20  reported  having  used  risk 
assessment  techniques  that  could  be  called 
formal  and  quantitative.  The  majority  of 
the  PMOs  reporting  the  use  of  formal , 
quantitative  techniques  were  from  the  Army. 

In  fact,  12  of  the  18  Army  PMOs  contacted 
reported  the  use  of  such  techniques.  The 
Air  Force  PMOs  contacted  were  much  more 
likely  to  use  qualitative  assessment 
methods  than  quantitative  with  the  Navy 
falling  between  the  two  other  Services. 


3  of  the  5  surveyed  in  the  SC 
category 

16  of  the  34  surveyed  in  the  PM 
category 

_ 1_  of  the  7  surveyed  in  the  PC 

category 

20 

It  was  four.d  that  of  the  20  PMOs 
reporting  that  quantitative  assessments  had 
been  used,  4  had  actually  accomplished  the 
assessments  within  the  PMO,  13  had  their 
assessments  done  by  functional  support 
groups  within  the  host  matrix  organization, 

2  used  a  support  contractor,  and  the  final 
1  used  a  PMO/Support  Contractor  team.  None 
had  used  their  prime  contractor  for  quanti¬ 
tative  assessments. 

The  survey  revealed  that  these  20  PMOs 
had  actually  made  a  total  of  47  applications 
of  formal,  quantitative  techniques.  The 
decision  categories  that  these  applications 
were  intended  to  serve  and  the  number  of 
applications  in  each  category  are  shown  in 
Figure  2  below. 

Decision  Category  No.  of  Applications 

Reported 


For  the  purposes  of  this  study  and  the 
handbook,  three  levels  of  program/project 
management  organization  were  defined.  At 
the  top  was  the  "Self-Contained"  (or  SC) 
Program  Management  Office,  an  organization 
frequently  staffed  by  more  than  one  hundred 
people  who  execute  all  the  major  functions 
of  program  management.  The  intermediate 
level  of  Program  Management  Office  strength 
was  called  the  PM.  This  was  Intended  to 
represent  an  office  of  less  than  100  people, 
who  carry  out  the  top  level  management, 
engineering,  and  business  functions.  This 
office  level  is  usually  supported  for  the 
majority  of  the  daily  detail  engineering, 
planning,  and  analysis  tasks  by  support 
organizations  within  the  host  command.  The 
lowest  level  of  PMO  strength  was  called  the 
PC  and  could  be  the  project  engineer, 
program  coordinator  or  product  manager. 

The  PC  represents  an  office  consisting  of 
one,  or  at  most,  three  people,  who  would 
usually  be  subsystem  development  engineers. 
The  host  command  generally  would  be  expected 
to  provide  support  to  these  offices, 
however,  comm:nd  priorities  often  leave 
this  PMO  level  with  little  in  the  way  of 
analytic  support. 

The  20  PMOs  that  reported  using  quanti¬ 
tative  techniques  were  made  up  of  the 
following; 


Tech.  Alternative  Selection  11 
Planning  14 
POM  Development/Budgeting  14 
Source  Selection  1 
Acquisition  Strategy  5 
Management  Control  2 
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During  all  the  interviews  conducted  in 
this  survey  an  attempt  was  made  to  deter¬ 
mine  the  highest  level  within  the  chain  of 
command  at  which  results  of  the  risk  assess¬ 
ment  were  reviewed.  The  highest  review 
level  for  the  47  applications  of  formal, 
quantitative  techniques  is  shown  in  Figure 
3  below. 


Highest  Level 
of  Review 


No.  of  Applications 
Reported 


PMO  22 
SYSCOM  5 
MATCOM  3 
SERVICE  HQ.  13 
OSD  4 
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B .  Techniques  8einq  Used 

The  survey  cf  the  PMOs  and  the  asso¬ 
ciated  support  offices  revealed  that  the 
formal,  quantitative  techniques  actually  in 
use  could  be  grouped  into  the  following  4 
categories :* 

-  Network  Analysis 

-  Decision  Analysis 

-  Estimating  Relationships 

-  Risk  Factors 

These  and  three  other  techniques  are  des¬ 
cribed  in  some  detail  in  the  handbook. 

Since  the  primary  purpose  of  this  paper  is 
to  discuss  the  findings  of  the  research 
rather  than  to  describe  how  to  conduct  a 
risk  assessement,  the  techniques  will  only 
be  briefly  described  here. 

The  networking  techniques  identified 
during  the  survey  were  of  the  classical 
type  that  have  emerged  from  the  PERT 
approach.  -In  most  cases  computer  simulation 
techniques  were  being  used  to  develop 
probability  statements.  Network  models 
were  found  in  use  throughout  the  Army  to 
support  program  milestone  decisions  at  In- 
Process  Reviews,  ASARCs  and  DSARC  reviews. 
Some  Navy  program  offices  were  found  to  be 
using  network  -  based  approaches  to  conduct 
risk  assessments  and  were  taking  them  one 
step  further  to  serve  as  program  control 
and  management  systems.  This  application 
will  be  addressed  in  more  detail  in  the 
RECOMMENDATIONS  section  of  this  paper. 

The  category  designated  as  decision 
analysis  techniques  consists  of  the  approach¬ 
es  that  examine  overall  program  decisions 
by  breaking  them  into  sequences  of  more 
elemental  decisions  with  their  associated 
uncertain  occurrences.  The  resulting 
sequences  often  are  depicted  as  decision 
trees,  but  other  approaches  such  as  proba¬ 
bilistic  event  analysis  are  also  possible. 

The  two  applications  found  during  preparation 
of  the  handbook  followed  the  concept  of  the 
decision  tree  approach  and  established 
schedule  and  cost  probability  distributions 
around  anticipated  problems  and  then 
utilized  a  computer  model  to  perform  simu¬ 
lation  exercises.  These  applications  were 
used  to  aid  the  planning  process  in  Army 
programs. 

The  estimating  relationships  technique 
is  used  by  Air  Force  Commands  to  develop 
estimates  of  management  reserves  necessary 


*The  handbook  actually  addresses  three 
other  techniques  which  have  potential  * 
for  use  hv  the  defense  program  manager. 


to  accommodate  contract  cost  risks  in  much 
the  same  manner  as  cost  estimating  relation¬ 
ships  are  used  in  parametric  cost  estimating. 
Based  on  historical  data,  curves  have  been 
developed  that  display  the  appropriate 
management  reserve  corresponding  to  various 
judgementally  assessed  levels  of  the 
following  factors  -  engineering  complexity, 
degree  of  system  definitization ,  contractor 
proficiency/experience  and  multiple  users. 

To  use  this  technique,  a  contract  is  des¬ 
cribed  by  program  management  personnel  and 
rated  relative  to  the  existence  of  these 
factors.  The  existing  curves  are  then  used 
to  identify  recommended  management  reserves. 

The  risk  factor  method  is  also  used  to 
estimate  cost  risks.  This  method  starts 
with  a  work  breakdown  structure  of  program 
elements.  Baseline  cost  estimates  fBCE) 
are  developed  for  each  program  element. 

Risk  factors  for  each  element  are  then 
developed  using  any  of  the  judgemental 
approaches  described  in  (4).  These  risk 
factors  are  then  multipled  times  the 
program  element  BCE  to  determine  the  amount 
to  be  added  to  the  budget  for  accommodating 
technical  risk.  This  is  the  technique  most 
commonly  used  within  the  Army  to  develop 
TRACE  budget  estimates. 

Figure'  4  on  the  next  page  -eports  the 
number  of  times  a  particular  technique  was 
reported  in  use  during  the  survey,  catego¬ 
rized  both  by  the  technique  type  and  by  the 
program  activity  being  supported  by  the 
application  of  that  technique.  It  can  be 
seen  that  networks  were  most  often  used  and 
were  largely  applied  to  assisting  the  se¬ 
lection  of  program  technical  alternatives 
and  for  program  planning.  The  risk  factor 
technique  was  the  second  most  used  technique. 
Its  primary  use  was  to  assist  budgeting 
decisions,  specifically  the  development  of 
TRACE  estimates. 

C.  Reported  Advantages  and  Disadvantages 

This  section  will  attempt  to  report  and 
summarize  various  statements  and  opinions 
received  during  'he  survey  as  a  result  of 
asking  the  users  to  describe  the  advantages 
and  disadvantages  of  the  various  techniques. 
Very  often  these  user  comments  are  focused 
on  the  resources  required  to  use  a  particular 
technique,  but  other  comments  are  also 
included. 

1 )  Networking  Techniques  -  The  net¬ 
working  techniques  in  use  were 
found  to  require  up  to  6  man- 
months  of  analyst  effort  at  the 
GS-12  to  GS-14  level  to  build  the 
program  network  for  a  large 
program  of  the  type  having  a 
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designated  program  manager 
(PM  type  program).  Much  of  this 
time  was  consumed  in  defining 
activity  inter-relationships  and 
gathering  experts'  estimates 
of  activity  risk.  In  nearly  every 
case,  this  effort  was  more 
difficult  and  time  consuming  than 
orginally  anticipated.  Several 
managers  reported  that  the  time 
required  to  build  the  network 
created  a  "catch-22"  situation 
since  the  longer  it  took  it  create 
the  network  and  complete  the 
analysis,  the  more  changes  that 
occurred  in  the  program,  in  turn, 
requiring  more  time  to  correct 
the  network.  A  few  managers 
reported  that  the  networks  and 
analyses  became  so  complicated  and 
took  so  long  that  they  were  not 
completed  in  time  to  support  the 
decision-making  process.  In 
these  cases,  the  assessments 
turned  out  to  be  little  more  than 
paper  exercises  used  to  sub¬ 
stantiate  decisions  already  made. 

Although  numerous  comments  were 
received  that  objected  to  the 
time  and  resources  required  to  use 
the  networking  techniques,  it  was 
also  acknowledged  that  the  by¬ 
products  of  the  process  -  e.g. 
better  program  planning,  definition 
of  interrelationships  and  more 
complete  understanding  of  the 
whole  program  and  its  parts  -  were 
extremely  valuable.  It  was  also 
reported  that  the  results  of  the 
network  analyses  had  a  relatively 
high  degreeof  credibility  with 
analysts,  managers  and  reviewers. 


2)  Decision  Analysis  Technique  - 
While  there  are  various  levels  of 
detail  and  rigor  that  can  be 
applied  in  using  the  methods  that 
fall  in  this  category,  the  only 
applications  found  during  the 
research  survey  tended  toward  a 
rather  sophisticated  approach. 
These  applications  required 
analysts  essentially  at  the  same 
level  and  for  the  same  duration  as 
the  network  technique  would 
require  for  a  comparable  program. 
Again,  the  collection  of  the 
necessary  program  data  to  provide 
as  input  to  the  technique  was  the 
most  difficult  and  time  consuming 
task. 

Although  Improved  program  planning 
was  gained  as  a  by-product  to  the 
actual  risk  assessment,  the 
detailed  understanding  of  activity 
sequencing  and  interrelationships 
that  result  from  the  development 
of  a  program  network  is  not  as 
direct  a  result  with  the  decision 
analysis  technique. 

3)  Estimating  Relationships  -  The 
time  consuming  part  of  the  effort 
associated  with  this  technique  was 
the  collection  and  analysis  of  the 
historical  data  necessary  to 
develop  the  contract  "factor"/ 
management  reserve  relationship. 
Once  established,  this  relation¬ 
ship  allowed  the  estimate  of 
management  reserves  by  mid-level 
analysts  with  only  a  few  days  of 
interviews  of  program  personnel. 

As  currently  applied,  this  method 
has  been  used  only  for  establish¬ 
ing  reserves  for  specific 
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contracts  not  for  program  cost 
estimates.  This  technique  is,  of 
course,  not  technically  rigorous 
since,  unlike  the  technique  of 
cost  estimating  relationships,  the 
"data"  used  to  establish  the 
relationships  are  subjective  and 
are  not  comparable  in  terms  of 
scales  of  measurement  and  are  not 
•even  scalable. 

4)  Risk  Factors  -  In  this  technique, 
as  with  all  the  others,  the 
collection  of  input  data  was  the 
most  demanding  task.  With  the 
risk  factor  method,  mid-level 
analysts  can  develop  the  necessary 
estimates  of  risk  for  each  program 
element  cost  through  interviews 
with  program  management  personnel. 
There  are  several  different 
approaches  suggested  for  gene¬ 
rating  these  risk  factors. 

However,  all  of  them  can  be  used 
to  develop  reasonable  estimates  in 
a  relatively  short  time.  This 
method  is  very  straight-forward 
and  understandable  and  thus, 
holds  significant  appeal  for 
many  management  personnel. 

The  point  estimate  of  a  risk 
factor  for  a  particular  element 
cost,  of  course,  tends  to  ignore 
the  uncertainty  existing  in  the 
defense  acquisition  environment. 
Again,  the  advantages  afforded  by 
the  networking  techniques  of 
improved  program  planning  and 
understanding  are  lacking  in  the 
risk  factor  method. 

CONCLUSIONS 

Many  of  those  familiar  with  this  re¬ 
search  effort  were  surprised  with  the 
finding  that  as  many  as  2D  of  the  46  PMOs 
surveyed  reported  that  they  had  employed 
formal,  quantitative  risk  assessment  tech¬ 
niques.  It  had  been  expected  that  fewer  PMOs 
would  have  used  these  techniques  and  that 
there  would  have  been  fewer  than  the  reported 
47  total  applications  supporting  the  6 
decision  categories  (Figure  2).  Despite 
this  better-than-expected  showing  for  the 
use  of  quantitative  assessments,  if  the 
survey  that  was  conducted  is  representative 
of  the  state  of  the  rest  of  the  defense 
acquisition  community,  there  remains  sub¬ 
stantial  room  for  improvement. 

In  the  face  of  encouragement  from  high 
levels  to  conduct  quantitative  risk  assess¬ 
ments,  such  as  the  AIP  action  that  Inspired 


this  research  effort,  and  the  basic  scien¬ 
tific  logic  that  instills  the  belief  that 
quantitative  is  "better"  than  qualitative, 
why  is  it  that  fewer  than  half  of  the  defense 
system  acquisition  programs  (according  to 
the  survey)  employ  formal,  quantitative  risk 
assessment  techniques?  The  authors  believe 
that  there  are  three  reasons  which  are 
supported  by  the  data  collected  during  this 
research  project. 

First,  there  must  be  interest  and 
support  for  risk  assessments  at  command 
levels  above  the  PMD.  It  is  obvious  without 
further  elaboration  -subordinate  activities 
will  concentrate  their  efforts  on  items  of 
interest  to  the  boss. 

Command  level  interest  and  support  of  a 
defense  acquisition  issue  is  typically 
demonstrated  by  the  issuance  of  policy  and 
procedure  directives.  AIP  Action  Number  11 
recommended  that  efforts  within  the  DoD  to 
quantify  risk  be  Increased  and  directed  that 
the  Services  adopt  TRACE  or  an  alternative 
concept  to  budget  funds  for  risk.  Despite 
this  emphasis,  the  most  recent  revisions  to 
the  primary  DoD  acquisition  policy  documents 
(DoDD  50DD.1  dtd  29  Mar  82  and  DoDI  5DD0.2 
dtd  8  Mar  83)  add  little  in  the  area  of 
requiring  the  quantification  of  program 
risks.  A  detailed  review  of  the  Service 
regulations  was  made  during  this  research 
project  to  locate  requirements  related  to 
risk  assessments.  Appendix  H  of  the  hand¬ 
book  consists  of  a  collection  of  excerpts 
from  these  Service  regulations.  While  these 
regulations  generally  encourage  the  use  of 
risk  assessments  at  various  points  within 
the  acquisition  process,  with  two  exceptions, 
none  of  them  clearly  state  the  need  for 
formal,  quantitative  assessments.  These  two 
exceptions  are  the  Army  directives  for 
implementing  TRACE.  (4)  and  (5)  This 
failure  of  the  system  of  directives  and 
regulations  to  specify  the  use  of  quanti¬ 
tative  techniques  indicates  to  the  authors 
and  more  importantly  to  the  program  managers 
a  lack  of  Interest  and  support  at  the 
highest  management  levels  within  DoD. 

Another  indicator  of  command-level 
interest  in  risk  assessments  is  the  regu¬ 
larity  at  which  the  assessments  that  are 
done  are  reviewed  by  these  higher  levels. 

As  was  shown  in  Figure  3,  the  data  collected 
in  this  survey  indicates  that  only  about  one 
half  of  the  assessments  completed  are 
reviewed  above  the  PMO  level.  Additionally, 
none  of  the  survey  respondents  reported  that 
any  review  of  a  formal,  quantitative  assess¬ 
ment  resulted  in  controversy  over  inputs, 
methods,  assumptions  or  results.  The 
contrast  of  this  situation  compared  to  that 


of  cost  estimates  is  so  great  that  a  lack 
of  interest  and  concern  about  the  results  of 
formal ,  quantitative  assessments  at  the 
higher  DoD  management  levels  is  clearly 
indicated. 

The  second  reason  why  more  acquisition 
programs  don't  employ  formal,  quantitative 
techniques  is  that  the  program  manager  must 
first  believe  that  there  is  "something  in  it 
for  him"  and  in  many  cases,  in  today's 
environment,  there  is  little  reason  for  him 
to  hold  this  belief.  For  example,  the 
program  manager  is  under  continual  pressure 
from  various  sources  to  "buy  in"  to  overly 
optimistic  schedules.  He  may  be  well  aware 
of  the  risks  of  doing  so  and  even  may  be 
able  to  quantify  them,  but  this  knowledge 
usually  carries  little  weight  when  compared 
to  user  requirements  and  budgetary  con¬ 
straints.  If  the  results  of  risk  assess¬ 
ments  are  largely  ignored,  then  program 
managers  are  not  going  to  waste  management 
attention  on  them,  and  will  perform  then 
perfunctorily  at  best. 

Another  example  results  when  assessing 
cost  risks.  At  every  level,  the  review  of  a 
program  manager  budget  appears  to  be  an 
effort  to  force  him  to  reduce  his  budget. 

The  quantification  of  his  cost  risks  and  the 
subsequent  identification  of  funds  to 
accommodate  this  risk  is  viewed  by  program 
managers  as  an  open  invitation  to  budget 
reviewers  to  make  cuts.  This  occurs  at  all 
levels  in  the  review  process  and  can  even 
happen  at  the  highest  level  as  evidenced  by 
Congressional  action  on  the  Navy's  FY  83 
budget.  As  acknowledged  by  the  House 
Appropriations  Committee  (HAC)  in  their 
report  on  the  FY  83  DoD  Appropriation  Bill 
(7),  the  Navy  responded  to  AIP  Action  Number 
11  by  Identifying  funds  required  for  manage¬ 
ment  reserves  in  several  R,D,T4E  and  Pro¬ 
curement  program  budget  requests.  The  HAC 
recommended  reductions  in  these  management 
reserves  totalling  over  $500  million  based 
on  their  determination  that  "the  factors 
being  used  by  the  Navy  in  arriving  at  the 
fund  requests  for  these  cost  elements  were 
not  appropriate  and  reasonable;  the  funds 
provided  were  being  used  for  other 
than  purposes  intended,  and  the  newly 
established  management  reserves  are  creating 
excessive  program  management  flexibility  or 
another  source  of  funding  that  can  be  drawn 
on  as  the  needs  of  a  program  arise."  (7 
p.  101)  While  the  approach  taken  by  the 
Navy  in  calculating  and  presenting  the 
requests  for  these  reserves  may  be  ques¬ 
tioned,  the  fact  remains  that  this  is 
another  example  of  funds  budgeted  for 
management  reserves  once  identified, 


becoming  very  vulnerable.*  Army  programs 
using  the  TRACE  approach  for  budgeting  for 
risk  have  been  much  more  successful  in 
sustaining  their  risk  estimates  in  their 
budgets  and  those  budgeting  efforts  were,  in 
fact,  acknowledged  in  the  HAC  reports.  This 
will  be  further  addressed  in  the 
RECOMMENDATIONS  section  of  this  paper. 

A  third  reason  why  more  system  acqui¬ 
sition  programs  don't  use  formal,  quantita¬ 
tive  risk  assessment  techniques  is  that  the 
techniques  don't  appear  "useable"  to  the 
program  manager.  The  term  "useable"  is 
intended  to  include  the  concepts  that  the 
techniques  are  neither  timely,  convincing 
nor  affordable. 

For  a  risk  assessment  to  be  "timely"  it 
must  provide  its  output  at  a  point  in  time 
that  will  allow  the  manager  to  analyze  the 
output,  assess  his  alternatives  and  make  the 
required  decision  within  the  time  allotted. 
Several  of  the  program  management  personnel 
interviewed  during  the  survey  indicated  that 
many  of  the  issues  they  were  dealing  with 
would  have  benefitted  from  risk  assessments, 
but  there  wasn't  sufficient  time  to  conduct 
such  an  assessment  and  still  meet  the  dead¬ 
line  for  the  decision.  There  were  also 
cases  reported  where  the  data  collection 
portion  of  the  assessment  effort  task  took 
so  much  longer  than  originally  anticipated 
that  the  assessments  were  completed  too  late 
to  support  the  decision-making  process. 

The  need  for  the  risk  assessment  tech¬ 
nique  to  be  "convincing"  comes  from  the 
program  manager's  desire  to  use  the  results 
of  the  assessment  to  influence  decisions 
made  at  levels  above  him.  For  a  technique 
to  be  "convincing"  it  must  be  theoretically 
sound  and  most  importantly,  understandable 
and  logical  to  the  reviewers  approving  the 
decision.  This  says  that,  no  matter  how 
sophisticated  the  actual  analysis  may  be¬ 
come,  for  a  technique  to  be  considered 
convincing,  it  must  be  explainable  in  terms 
understood  by  managers  and  reviewers  who  may 
be  totally  unfamiliar  with  risk  assessment 
concepts. 

The  need  for  a  risk  assessment  technique 
to  be  "affordable"  refers  to  the  availabi¬ 
lity  of  personnel  with  the  necessary  capabi¬ 
lities  for  the  time  period  required,  whether 
these  people  come  from  the  PMO  itself,  the 
functional  support  groups  of  the  host 


*It  should  be  noted  that  a  portion  of  the 
management  reserves  requested  by  the  Navy 
(for  the  TRIDENT  and  CV-SLEP  programs) 
were  eventually  restored  by  the  Conference 
committee.  (8  p.  127) 
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command  or  a  contractor.  This  concept  of 
"affordability"  includes  the  analysts 
involved  in  conducting  the  assessment  as 
well  as  the  individuals  within  the  PMO  and 
elsewhere  who  will  serve  as  the  subject 
matter  experts  and  who  will  fulfill  the 
critical  function  of  providing  the  input 
data.  In  addition,  "affordability"  of  a 
risk  assessment  must  consider  the  availabi¬ 
lity  of  the  program  manager  himself,  who 
must  be  involved  to  give  direction  to  the 
assessment,  to  agree  to  necessary  simplifying 
assumptions,  to  make  critical  choices  as 
the  assessment  progresses  and  to  apply  the 
results  of  the  assessment.  In  several  cases 
during  the  survey,  program  managers  indicated 
that  this  issue  of  "affordability"  was  the 
most  critical  problem  for  them.  This  was 
particularly  true,  of  course,  with  the  more 
sophisticated  techniques  such  as  networking 
and  decision  analysis. 

RECOMMENOATIQNS 

The  recommendations  resulting  from 
this  study  are  aimed  at  several  different 
levels  within  the  defense  acquisition 
management  community.  For  the  policy¬ 
making  levels  within  the  Office  of  the 
Secretary  of  Defense  and  the  Services,  it 
is  hoped  that  they  will  recognize  that  the 
perception  of  their  attitude  is  critical 
for  setting  the  tone  for  all  the  risk 
assessment  activities  that  occur  within 
DoO.  This  perception  is  developed  from  the 
policy  directives  and  regulations  that  are 
issued  and  the  extent  of  interest  demon¬ 
strated  in  reviewing  the  results  of  risk 
assessments.  Thus,  the  specific  recom¬ 
mendations  in  this  area  are:  1)  revise  the 
tree  of  appropriate  directives  and  regula¬ 
tions  (starting  with  000D  5000.1  and  0001 
5000.2  and  continuing  through  the  Service 
regulations)  to  explicitly  require  that 
formal ,  quantitative  assessments  be  conducted 
arid  under  what  circumstances  they  are 
required  and  2)  critically  review  and 
utilize  the  results  of  such  risk  assessments 
as  decisions  are  made  at  appropriate  points 
in  the  fiscal  and  life  cycles  of  programs. 

One  of  the  conclusions  discussed  in 
the  preceding  section  promotes  the  concept 
that  a  program  manager  will  not  expend  the 
resources  necessary  to  conduct  a  formal , 
quantitative  risk  assessment  unless  "he 
sees  something  in  it  for  him"  or  for  his 
program.  If  Increasing  requirements  to 
conduct  risk  assessments  are  going  to  be 
placed  on  program  managers,  then  procedures 
must  be  clearly  established  so  that  the 
various  review  points  recognize  and  honor 
the  estimates  of  risk.  This  does  not  mean 
that  estimates  resulting  from  a  quantitative 
assessment  should  be  accepted  without 


questioning  the  technique.  To  the  contrary, 
all  elements  of  the  assessments,  that  is, 
the  input,  techniques,  assumptions  and  the 
conclusions  should  be  questioned  and  explored. 

What  this  does  mean  however,  is  that 
assuming  it's  determined  that  a  reasonable, 
justifiable  approach  has  been  taken  in 
developing  the  risk  estimates,  the  funding 
requirements  then  identified  to  accomodate 
the  risk  (in  the  case  of  cost  risk)  should 
not  be  arbitrarily  viewed  as  the  discretio¬ 
nary  budget  line  that  is  the  first  to  be 
cut  when  it  comes  time  to  squeeze  more 
programs  into  the  overall  budget. 

Congress,  of  course,  is  one  of  the  key 
review  points  that  must  agree  and  be  part 
of  any  attempt  to  increase  the  use  of 
quantitative  assessments.  At  the  same  time 
that  it  was  recommending  reductions  to 
remove  management  reserves  from  the  Navy's 
FY83  ROT&E  and  Procurement  requests,  the 
HAC  acknowledged  the  "need  for  management 
reserves  to  provide  the  necessary  flexibility 
to  address  unforeseen  circumstances." 

While  they  addressed  the  need  for  reserves, 
the  HAC  also  reported  that  they  "found 
little  evidence  to  support,  or  otherwise 
validate  the  appropriateness  and  reason¬ 
ableness  of  the  factors  being  used,  or  the 
resultant  amounts  being  included  in  the 
Navy's  budget  requests."  (7  p.  101) 

Clearly,  the  Services  and  OSD  should  develop 
a  logical,  justifiable  approach  to  assessing 
risks,  estimating  the  cost  associated  with 
these  risks,  presenting  the  budget  require¬ 
ments  necessary  to  accommodate  these  risks 
and  managing  the  funds  appropriated  for 
these  risks.  This  approach  must  then  be 
negotiated  with  the  key  committees  in 
Congress. 

This  recommended  approach  would  seem 
to  be  so  simple  and  straight-forward  that 
It  would  appear  that  it  must  be  naively 
overlooking  some  Insurmountable  obstacles. 
However,  this  is  basically  the  approach 
that  the  Army  has  taken  in  using  the  TRACE 
concept  to  budget  for  risk  in  selected 
ROT&E  and  now  Production  programs.  As 
viewed  by  the  Army  program  manager,  the 
TRACE  concept  certainly  "has  something  in 
it  for  him"  since  his  estimates  for  technical 
risk  are  routinely  funded  in  the  Army 
budget  process. 

The  final  reason  discussed  in  CONCLUSIONS 
for  the  limited  application  of  formaT"! 
quantitative  techniques  was  ‘he  need  for 
the  program  manager  to  find  the  technique 
"useablt".  To  provide  assistance  in  this 
area  is  a  real  challenge  to  the  developers 
of  these  techniques.  From  the  data  collected 
during  the  survey,  it  is  clear  that  acqui- 


sition  management  personnel  have  at  least 
some  appreciation  for  the  value  of  the 
information  that  can  result  from  implementing 
the  more  sophisticated  techniques  of 
networking  and  decision  analysis.  But, 
because  of  the  resources  (dollars,  people 
and  time)  required,  these  techniques  are 
only  well-suited  for  major  decisions  that 
can  be  anticipated  far  in  advance  on  resource- 
rich  programs. 

A  number  of  recommendations  evolve 
from  this  conclusion.  First,  the  net¬ 
working  techniques  in  use  should  be  extended 
to  perform  on-going  program  control  functions 
in  addition  to  risk  assessments.  Once  the 
time  and  effort  has  been  invested  to  create 
the  original  network  and  to  apply  computer 
software  in  order  to  conduct  the  risk 
assessment,  this  effort  should  be  capitalized 
on  by  maintaining  the  network  for  program 
control.  Commercially  available  software 
packages  have  been  used  in  this  manner  on 
some  Navy  shipbuilding  programs. 

Another  recommendation  in  this  area  is 
to  educate  program  managers  and  upper  level 
managers  throughout  the  community  on  the 
very  real  and  valuable  by-products  to  be 
gained  from  employing  any  quantitative 
technique  -  especially  the  more  rigorous 
network  and  decision  analysis  techniques. 

These  by-products  ••  such  as  improved  program 
planning,  identification  and  recognition  of 
responsibility  for  all  program  activities 
and  better  understanding  of  the  sequencing 
of  these  activities  -  in  many  cases  may  be 
much  more  important  then  the  originally 
stated  purpose  of  generating  an  estimate  of 
program  risk.  This  is  presently  not  under¬ 
stood  or  appreciated  by  managers  at  many 
levels  within  the  DoD. 

Where  the  availability  of  dedicated, 
analytical  support  is  the  primary  factor 
limiting  the  conduct  of  risk  assessments, 
upper  level  management  should  consider  the 
establishment  of  such  offices  within  the 
functional  support  matrix.  While  this  can 
be  expensive  (in  terms  of  both  dollars  and 
billets)  there  are  many  advantages  to 
developing  an  "in-house"  capability  for 
conducting  program  sensitive  risk  assessments. 

The  final  recommendation  centers  on 
encouraging  the  development  of  techniques 
that  the  manager  of  any  size  program  can 
find  "usable"  to  assist  the  decisions  he 
must  make  with  advance  warning  of  perhaps 
days  rather  than  months.  Techniques  that 
are  timely,  easy  to  understand,  convincing 
and  affordable  are  desperately  needed.  To 
become  convinced  that  such  techniques  will 
be  used,  if  available-,  one  need  only  look 
at  the  number  of  times  the  risk  factor 


method  has  been  employed  to  de»elop  TRACE 
estimates . 
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TOTAL  RISK  ASSESSING  COST  ESTIMATE  FOR 
PRODUCTION  (TRACE-P): 

WILL  HISTORY  REPEAT  ITSELF? 

by  Paul  E.  Grover 

US  Army  Logistics  Management  Center 


A  key  milestone  in  the  life  cycle  of  an 
Army  weapon  system  is  the  decision  to  enter 
production.  Milestone  III— the  production 
decision  point— occurs  years  after  concept 
exploration  begins,  but  only-a  small  per¬ 
centage  of  the  total  expected  program  life 
cycle  cost  has  been  incurred  up  to  this  point. 
Most  of  the  program  funds  remain  to  be  expend¬ 
ed  during  the  Production/Deployment  Phase 
following  Milestone  III  and  in  the  operational 
phase.  Typically,  R&D  accounts  for  approxi¬ 
mately  15%  of  the  program  cost,  production 
30%,  and  operation  and  support  the  remaining 
55%. [1]  Therefore,  it  is  prudent  that  the 
risks  of  entering  production  be  understood  as 
clearly  as'  possible  before  the  program  is  com¬ 
mitted  further. 

In  spite  of  the  Army's  efforts  to  prepare 
for  production  and  to  control  cost  through 
Producibility,  Engineering  and  Planning  (PEP) 
and  Production  Readiness  Reviews  (PRR),  a 
weapon  system  typically  experiences  problems 
during  the  transition  period  from  R&D  into 
Production  which  eventually  contribute  to  sub¬ 
stantial  increases  in  program  cost.  Figure  1 
shows  the  extent  and  alarming  trend  of  the 
total  increase  in  RDT&E  and  Procurement 


appropriations  for  Army  systems  reported  in 
the  Selected  Acquisition  Report  (SAR).[2] 
Excluding  inflation,  there  still  has  been  a 
28%  increase  in  the  procurement  appropriation 
from  the  first  quarter  of  FY  78  through  the 
third  quarter  of  FY  81  compared  to  15%  in¬ 
crease  in  RDT&E  during  the  same  period. 

Although  DOD  may  attempt  to  shift  produc¬ 
tion  risk  to  a  defense  contractor  through  the 
use  of  firm,  fixed  price  (FFP)  contracts,  the 
Army  Procurement  Research  Office  (APRO)  has 
shown  in  earlier  work  that  this  has  little  im¬ 
pact  on  cost  growth.  "FFP  contracts  suffered 
a  net  53%  cost  growth  -  almost  identical  to 
the  entire  sample  cost  growth. "[3]  Nearly  all 
of  the  FFP  contracts  were  for  production.  The 
need  for  more  attention  to  production  and  for 
better  planning  and  control  of  initial  produc¬ 
tion  uncertainties  is  clear.  Also,  a  method¬ 
ology  is  needed  to  analyze  the  uncertainties 
of  entering  production  and  to  account  for  the 
risk  impacts  in  the  program  plan.  By  planning 
and  budgeting  for  risks,  program  disruptions 
and  resulting  cost  increases  can  be  minimized. 

The  magnitude  and  trend  of  transition 
problems  have  been  studied  in  terms  of  budget¬ 
ary  impact.  In  1979,  Augustine  reported  that, 
for  38  DOD  programs  from  1962-1976,  program 
cost  growth  of  9%  occurs  after  R&D  is  complete, 
adjusting  for  inflation  and  quantity  changes^] 
Factoring  out  the  R&D  cost,  procurement  cost 
growth  for  these  SAR  programs  was  about  12%. 

To  evaluate  current  programs,  APRO  and  the 
Army  Logistics  Management  Center  (ALMC) 


FIGURE  1  uan  WEAPON  ST5TEK  COST  GJWTX 
(SOURCE:  STUOT  OF  AJUW  LOGISTICS.  1961.  AM  AJWY  SAJI'S) 
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analyzed  11  Army  SAR  systems  that  have  recent¬ 
ly  undergone  transition  from  R&D  to  production 
to  determine  procurement  cost  growth  during 
early  production. [5]  SAR  Procurement  cost 
data  in  constant  dollars  was  adjusted  for 
quantity  changes  with  a  baseline  taken  at  the 
quarterly  SAR  three  to  six  months  prior  to 
ASARC  III.  The  magnitude  of  growth,  averaging 
35 . 5%  and  totaling  over  $5  billion,  indicates 
an  unfavorable  trend  of  higher  growth  in  re¬ 
cent  years  as  shown  in  Figure  2.  Most  of  the 
overall  35 . 5%  growth  occurs  in  the  first  24 
months  of  the  time  period  considered,  with 
very  steep  rises  between  3-9  months  (ASARC  III 
time  frame)  and  18-24  months  (1-1  1/2  years 
after  ASARC  III).  The  timing  of  reported 
growth  seems  to  correlate  with  the  updated 
cost  estimates  prepared  for  ASARC  III  (t  = 

6  months)  and  the  annual  contract  negotiation 
and  award  cycles. 


Multi-year  Procurement,  #4  -  Program  Stability, 
#5  -  Capital  Investment,  #7  -  Economic  Produc¬ 
tion  Rates,  #8  -  Appropriate  Contract  Type, 

#10  -  Reduce  Administrative  Cost  and  Time, 
#13-14  -  To  Reduce  "Red  Tape,"  #15  -  Funding 
Flexibility,  #18  -  Budget  for  Inflation,  #19  - 
Forecasting  Business  Base,  #20  -  Improved 
Source  Selection,  #22  -  Design  to  Cost  and 
others  are  management  actions  that  will  reduce 
'and  control  risks  associated  with  the  transi¬ 
tion  of  systems  from  R&D  into  production. 
Others,  like  #6  -  Budget  to  Most  Likely  Cost, 
and  particularly  #11  -  Budget  Funds  for  Tech¬ 
nological  Risk,  are  designed  to  help  minimize 
the  adverse  impacts  of  problems  when  they  do 
occur.  In  requiring  the  services  to  budget 
funds  for  risk,  the  initiative  pointed  out: 

"Material  development  and  early  pro¬ 
duction  programs  are  subject  to  uncertain- 
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The  cost  growth  pattern  exhibited  by  the 
aggregate  of  the  11  systems  is  not  evident  when 
analyzing  each  individual  system.  Some  pro¬ 
grams  grow  from  the  beginning,  others  delay 
growth  until  later  in  the  production  cycle, 
and  others  have  experienced  little  growth. 

This  is  consistent  with  the  notion  of  uncer¬ 
tainty  and  probabilistic  occurrences.  Statis¬ 
tical  techniques,  although  applied  to  individ¬ 
ual  programs,  are  more  meaningful  on  an  aggre¬ 
gate  basis. 

To  address  this  problem  and  some  other 
shortcomings  in  the  DOD  acquisition  process, 
on  30  April  1981,  the  DEPSECDEF,  Mr.  Carlucci, 
directed  a  series  of  initiatives  to  change  for 
the  better  some  of  the  policies  of  the  past. 
Such  Initiatives  as  #1  -  Management  Principles, 
#2  -  Pre-Planoed  Product  Improvement,  #3  - 


ties.  Program  managers  who  explicitly 
request  funds  to  address  these  uncertain¬ 
ties  usually  find  these  funds  deleted 
either  in  the  DOD  PPBS  process,  by  0MB,  or 
by  Congress.  Then,  when  such  uncertain¬ 
ties  occur,  undesirable  funding  adjust¬ 
ments  are  required  or  the  program  must  be 
delayed  until  the  formal  funding  process 
can  respond  with  additional  dollars. 

The  Army  has  initiated,  and  Congress 
has  accepted,  a  Total  Risk  Assessing  Cost 
Estimate  (TRACE)  to  explicitly  address 
program  uncertainties  in  the  development 
of  RDT&E  budget  estimates.  The  Army  is 
studying  the  application  of  this  concept 
to  early  production  cost  estimates.  The 
other  Services  lack  a  similar  concept  to 
justify  reserve  funds  for  dealing  with 
developmental  uncertainties." 
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Shortly  thereafter,  in  a  Vice  Chief  of 
Staff  Army  Memorandum,  22  Jul  81,  on  Cost 
Discipline,  a  directive  to  implement  the  con¬ 
cept  of  a  management  reserve  fund. for  procure¬ 
ment  with  emphasis  on  the  transition  phase  be¬ 
tween  engineering  and  production  was  issued. 
Thus,  the  Army  started  a  program  to  budget 
funds  for  production  risks  and  included  such 
funds  for  programs  like  the  Remotely-Piloted 
Vehicle,  Apache  and  other  programs  at  subse¬ 
quent  ASARC  reviews.  The  process  was  formal¬ 
ized  on  6  Oct  82  when  the  Comptroller,  US  Army 
Materiel  Developments  and  Readiness  Command 
(DARCOM)  issued  a  Letter  of  Instruction  (LOI) 
on  Total  Risk  Assessing  Cost  Estimate  for  Pro¬ 
duction  (TRACE-P).  Therefore,  the  Army  has 
adopted  policies  that  seek  to  reduce,  control 
and  manage  risk  in  R&D  and  production,  with 
the  knowledge  from  experience  that  even  the 
best  managed  program  may  develop  foreseen  and 
unforeseen  circumstances  that  will  increase 
cost.  Funding  for  these  risks  will  permit 
timely  resolution  and  minimize  resultant  cost 
increases. 

PURPOSE  AND  OBJECTIVE 


point  estimate  selected  from  that  distribution 
such  that  its  probability  of  being  exceeded  is 
at  an  acceptable  level.  Figure  3  shows  the 
TRACE  as  having  an  equal  (50/50)  probability 
of  underrun  and  overrun.  When  the  Baseline 
Cost  Estimate  (BCE)  is  placed  on  this  distri¬ 
bution  of  possible  costs,  it  will  fall  to  the 
left  of  the  TRACE.  The  BCE  is  calculated  from 
engineering  estimates  of  specifically  pro¬ 
grammed  activities,  and  although  it  generally 
includes  some  contingency,  it  does  not  include 
consideration  for  many  uncertain  activities 
that  are  statistically  probable. 


PROBABILITY 


BCE  TRACE 


This  paper  is  designed  to  compare  the 
Army  efforts  of  TRACE  versus  TRACE-P.  In 
doing  so,  the  TRACE  process,  methodology,  his¬ 
tory  and  results  will  be  highlighted.  Sec¬ 
ondly,  the  evolution  of  TRACE-P  will  be  dis¬ 
cussed  with  primary  focus  on  an  APRO/ALMC 
study  effort  that  took  place  from  June 
through  December  1981.  Characteristics  of 
TRACE  and  TRACE-P  will  then  be  compared  and 
contrasted.  Based  on  the  experiences  of  TRACE 
during  its  early  years  (1976-1979)  and  the 
similarities  between  TRACE  and  TRACE-P,  some 
predictions  will  be  made  concerning  the  use  of 
TRACE-P  in  FY  84  and  beyond.  Thus,  the 
TRACE-P  program  will  be  presented  as  a  logical 
extension  of  TRACE  into  a  problem  area  of  high 
risk,  with  the  expectation  that  many  of  the 
experiences  with  TRACE  will  be  repeated  with 
TRACE-P. 

TRACE  FOR  R&D 

In  a  memorandum,  "RDTE  Cost  Real  ism, 11 
dated  12  July  1974,  Mr.  Norman  Augustine,  ASA 
(R&D)  formulated  the  TRACE  concept  to  provide 
realistic  cost  estimates  and  thereby  minimize 
subsequent  disruptive  reprogramming  actions. 

It  was  designed  to  minimize  cost  overruns  with¬ 
out  resorting  to  gross  overbudgeting.  Specific 
emphasis  is  on  allocation  of  funds  to  reduce 
the  cost  growth  effects  resulting  from  the 
occurrence  of  events  that  could  not  be  pro¬ 
grammed  because  of  the  lack  of  certainty  that 
they  would  materialize. [6] 

It  is  impossible  to  exactly  predict  the 
cost  of  a  project.  In  reality,  the  final  cost 
of  a  project  will  fall  somewhere  within  a  dis¬ 
tribution  as  shown  in  Figure  i.  TRACE  is  a 
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The  PMO  calculates  a  BCE  and  a  TRACE  and 
submits  them  through  channels  to  Headquarters, 
Department  of  the  Army  (HQDA).  The  BCE  repre¬ 
sents  project  target  cost.  The  TRACE  is  used 
for  progranming/budgeting  and  as  the  cost 
entry  in  the  Five  Year  Defense  Plan  (FYDP) . 

Upon  budget  approval  by  DOD/Congress,  the  BCE 
amount  is  released  to  the  PM  for  program  exe¬ 
cution.  The  remainder,  known  as  "TRACE  defer¬ 
ral  or  Risk  Cost,"  is  retained  by  HQDA  to  serve 
as  a  source  of  funds  for  the  PM  to  draw  upon, 
when  justified,  to  accomplish  additional  uncer¬ 
tain  activities. 

The  retention  of  TRACE  deferral  funds  by 
HQDA  allows  for  managerial  control  and  possible 
cost  savings.  These  deferral  funds  are  on  a 
line  item  and  fiscal  year  basis,  and  each 
year's  funds  are  available  for  obligation  for 
two  years.  During  the  fifth  quarter  of  avail¬ 
ability,  the  PMO  must  decide  whether  to  release 
the  funds  to  HQDA  for  reprogramming.  The  funds 
will  be  automatically  reprogrammed  if  no  action 
is  taken  by  the  seventh  quarter.  Any  request 
by  the  PM  to  obtain  deferral  funds  must  be 
accompanied  by  sound  justification  within 
established  guidelines.  For  example,  funds  will 
not  be  used  to  offset  costs  of  major  require¬ 
ments  changes  which  instead  will  be  accommo¬ 
dated  by  restructuring  the  program  and  recom¬ 
puting  the  TRACE.  The  list  of  exclusions  in¬ 
cludes  the  following  occurrences  for  which 
TRACE  deferral  funds  may  not  be  used: 


1.  Program  changes  based  on  requirements 
changes. 

2.  Inflation-related  cost  increases. 

3.  Congress ionally  enacted  pay  raises. 

A  survey  of  20  Army  PMO's  that  had  TRACE 
experience  found  that  three  principal  method¬ 
ologies  were  used  to  compute  TRACE,  with  some 
unique  variations. [7]  The  three  methods  were 
classified  as  Risk  Percentage,  Risk  Factor, 
and  Probabilistic  Network  Models.  A  fourth 
method  called  Probabilistic  Event  Analysis  or 
Risk  Tabulation  was  not  found  to  be  used  by 
any  PM.  These  four  methodologies  are  as 
follows: [8] 

1 .  Risk  Percentage 

The  Risk  Percentage  methcd  is  an  un¬ 
documented  procedure  in  which  the  TRACE  is 
computed  by  adding  a  percentage  (10-15?)  to 
the  BCE  at  a  summary  level.  The  percentage  is 
subjectively  determined  by  experts  based  on 
past  experience,  risk  assessment  and  judgment. 
Time  phasing  is  accomplished  subjectively  or 
assuming  proportionality  to  the  ROTE  effort. 
This  method,  although  used,  has  not  been  for¬ 
mally  approved  because  of  similarities  to  con¬ 
tingency  fund  or  management  reserve  concepts. 

2.  Risk  Factor 

The  Risk  Factor  approach  described  in 
a  1975  LOI  [9]  computes  TRACE  by  assigning  risk 
factors  to  discrete  Work  Breakdown  Structure 
(WBS)  elements  of  the  BCE.  After  estimating 
WBS  element  costs  as  part  of  the  RDTE  BCE, 
each  element  is  assessed  for  uncertainty  by 
experts.  Historical  data  on  previous  similar 
systems  is  used  when  available,  but  most  risk 
factors  are  subjectively  determined.  A  fact:  r 
represents  the  cost  increase  expected  for  a 
WBS  element  as  a  result  of  technical  uncer¬ 
tainty  associated  with  that  particular  element 
plus  the  interrelated  uncertainties  associated 
with  other  WBS  elements  that  interface  with 
it.  By  multiplying  each  WBS  cost  by  its  risk 
factor,  a  TRACE  Is  computed.  Time  phasing  is 
subjective  or  assumed  proportional  to  BCE  time 
phasing. 

Advantages  of  this  approach  are: 

(1)  analysis  does  not  require  a  high  ana¬ 
lytical  skill  level;  (2)  analysis  can  be 
performed  quickly  and  inexpensively  in  compar¬ 
ison  to  computer  modeling;  (3)  analysis  can 
be  easily  understood;  and  (4)  quality  of  anal¬ 
ysis  can  be  easily  evaluated  by  management. 

The  most  serious  disadvantage  of  this 
approach  is  in  the  determination  of  the  risk 
factors.  Because  of  the  apparent  simplicity 
of  the  approach,  there  might  be  a  tendency  to 
use  the  risk.factcr  as  simply  a  "fudge  factor." 


To  handle  the  factor  in  such  a  manner  would 
reduce  the  credibility  of  the  cost  estimate. 

In  addition,  the  factor  is  implicitly  assumed 
to  be  constant  for  each  element  throughout 
duration  of  the  project. 

3.  Probabilistic  Network  Modeling 

Probabilistic  Network  Modeling  is  a 
combined  approach  using  Program  Evaluation 
and  Review  Technique  (PERT)  principles  and 
Monte  Carlo  simulation  techniques.  Various 
computerized  models  can  be  used  for  this  ap¬ 
plication,  including  VERT,  RISCA,  and 
RISNET.[8]  An  R&D  program  is  first  displayed 
as  a  network  of  interrelated  events  and  ac¬ 
tivities.  Cost,  schedule  and  technical  un¬ 
certainties  associated  with  the  various  ac¬ 
tivities  are  then  estimated.  The  model  iter- 
ately  simulates  the  activities  and  events  to 
produce  time-phased  cost  and  schedule  distri¬ 
butions  for  the  program.  The  analyst  can 
adjust  the  TRACE  to  levels  reflecting  desired 
probability  of  cost  overrun  vs.  cost  underrun. 
This  method  is  the  most  rigorous  and  resource 
demanding,  but  it  is  the  most  precise  and  risk 
inclusive. 

Advantages  of  this  approach  are: 

(1)  explicit  consideration  is  given  to  activ¬ 
ity  interaction;  (2)  the  TRACE  may  be  selec¬ 
ted  from  the  total  cost  distribution  as  out¬ 
put;  (3)  the  form  and  collection  of  output  is 
flexible;  (4)  the  model  can  be  easily  modi¬ 
fied  and  rerun  to  answer  "what-if?"  questions; 
(5)  the  network  can  estimate  the  BCE  by  fix¬ 
ing  schedules  and  removing  uncertain/contin- 
gency/fallback  activities;  (6)  the  network 
car.  serve  as  a  management  tool  to  track  and 
control,  as  well  as  predict  time  and  schedule; 
and  (7)  the  network  can  be  used  to  satisfy 
the  Decision  Risk  Analysis  requirement. 

Disadvantages  of  this  approach  in¬ 
clude:  (1)  a  high  skill  level  is  required 
to  build  the  network  and  collect  data;  (2) 
the  output  can  be  sensitive  to  the  network 
logic;  (3)  it  is  difficult  to  reconcile  this 
approach  with  the  WBS;  (4)  the  cost  is 
initially  high;  and  (5)  it  requires  consider¬ 
able  data  collection. 

4.  Probabilistic  Event  Analysis 

Probabilistic  Event  Analysis,  or 
Risk  Tabulation,  was  developed  by  John  M. 
Cockerham  and  Associates,  Inc.  to  correct 
perceived  deficiencies  in  the  Risk  Factor 
approach[10].  Risk  for  each  WBS  element  is 
separated  into  two  categories  in  an  effort  to 
assess  interactive  effects  between  WBS  ele¬ 
ments.  Internal  (stand  alone  or  isolated) 
risks  are  a;sessed  as  well  as  the  external 
(interactive)  risks.  Using  conditional  prob¬ 
ability  theory,  the  overall  program  cost  risk 
is  tabulated.  Uncertainties  assessed  as 
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probability  values  are  determined  essentially 
in  the  same  manner  as  risk  factors.  Time 
phasing  can  be  incorporated  into  the  calcula¬ 
tion  by  estimating  when  the  various  risks  will 
occur  for  each  WBS. 

Advantages  of  this  approach  are  that 
it  is  relatively  easy  to  use,  and  it  addresses 
interaction  between  elements  so  it  should  give 
better  results  than  the  Risk  Factor  approach. 
Disadvantages  are  (1)  it  is  highly  dependent 
upon  the  skill  of  the  analyst  to  identify  and 
account  for  the.  various  interdependencies; 

(2)  it  is  sensitive  to  errors  in  subestimates 
and;  (3)  the  TRACE  uses  the  BCE  as  a  basis 
and  is  subject  to  the  same  bias  as  the  BCE. 

TRACE  for  RDT&E  has  been  applied  by  the 
Army  since  the  6  March  1975  "Letter  of  In¬ 
struction  (LOI)  for  Implementation  of  ROTE 
Cost  Realism  for  Current  and  Future  Develop¬ 
ment  Programs. "[9]  Between  FY  76  and  FY  84, 
TRACE  has  been  applied  to  31  programs.  As  of 
FY  81,  TRACE  has  been  credited  with  avoiding 
17  reprogramming  actions  and  avoiding  27  Con¬ 
gressional  approvals. [11]  The  number  of  pro¬ 
grams  using  TRACE  and  the  amount  of  TRACE 
funding  have  increased  as  shown  in  Figure  4. 
Those  programs  that  have  more  recently  used 
TRACE  have  relied  on  TRACE  as  a  greater  per¬ 
centage  of  RDTE  funding  in  the  TRACE-funded 
years  than  earlier  TRACE  programs,  as  shown 
in  Figure  5.  Thus  TRACE  for  R&D  has  become 
generally  accept°d  and  useful. 
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□  TRACE/RDTE  ON  ANHUAF  BASIS 
nan  s 


TRACE  for  R&D  has  experienced  some  prob¬ 
lems: 

1.  Initially,  PM's  were  hostile  to  the 
concept  of  project  funds  being  held  in  re¬ 
serve  by  DA.  [12] 

2.  In  theory,  approximately  one-half  of 
the  programs  using  TRACE  should  not  need  all 
of  their  programmed  TRACE  funds  and  the  other 
half  should  need  more  than  the  amount  pro¬ 
grammed.  In  reality  though,  very  few  (5 
times  out  of  88  potential)  programs  from  FY 
76-FY  81  have  turned  in  unused  TRACE  funds  in 
a  fiscal  year.  Two  possible  explanations  are 
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that  TRACE  may  have  been  historically  under¬ 
estimated  (or  underfunded)  or  that  TRACE  be¬ 
comes  a  "self-fulfilling  prophecy." 

3.  TRACE  methodology  has  been  criti¬ 
cized  as  too  subjective,  too  resource  demand¬ 
ing,  too  restrictive,  and  too  inflexible. [7] 

4.  TRACE  has  not  eliminated  RDTE  over¬ 
runs.  [12] 

TRACE-P 

As  stated  earlier,  TRACE-P  within  the 
Army  was  catalyzed  by  the  April  1981 
DEPSECDEF  memorandum  followed  by  a  July  1981 
Vice  Chief  of  Staff  memorandum.  The  first 
Army  effort  at  estimating  risk  funds  for  pro¬ 
duction  was  completed  in  the  spring  of  1981 
for  the  Remotely  Piloted  Vehicle  (RPV)  by  the 
Directorate  of  Plans  and  Analysis  at  the 
U.  S.  Army  Aviation  Research  and  Development 
Command.  A  methodology  for  estimating  man¬ 
agement  reserve  for  production  for  this  sys¬ 
tem  was  developed.  A  similar  approach  was 
used  for  the  Apache  helicopter  later  that 
summer.  Concurrently,  the  Army  Procurement 
Research  Office  began  a  formal  study  of 
TRACE-P  from  June  through  December  1981.  A 
methodology  similar  to  the  RPV  approach  was 
developed  and  tested  on  an  historical  system 
for  validation.  The  APRO  model  was  further 
modified  by  HQ  DARCOM  and  institutionalized 
in  the  6  October  1982  LOI.  Thus,  TRACE-P  has 
evolved  into  a  formal  policy  and  procedure. 

Capitalizing  on  the  experience  of  the 
TRACE  for  R&D,  the  mechanics  for  implementing 
TRACE-P  parallel  almost  identically  the  TRACE 
process.  In  the  6  October  1982  DARCDM  LOI, 
TRACE-P  is  defined  as  the  median  or  5D/5D 
point  on  the  cost  distribution  for  procure¬ 
ment  funds  during  the  first  three  years  of 
"significant  quantity  production."  Specif¬ 
ically  excluded  from  the  TRACE-P  are  procure- 
funds  for  long  lead  items  and  cost  increases 
caused  by: 

Qual ity  changes 

Performance  changes  to  meet  an  increased 
threat 

Poor  management 

Inadequate  funding  in  the  early  years 

Inflation  in  excess  of  rates  predicted 

Unknown-Unknowns 

Eight  risk  categories  are  allowed  to  con¬ 
trol  and  focus  the  analysis  of  risk  in  early 
production.  Cost  increases  covered  by  the 
TRACE-P  umbrella  are: 

Threat  uncertainty 

Management 

Material s/Purchaseci  Parts 

Facilities/Equipment 

Labor  * 


Design  changes 

Producibil ity 

Performance 

These  risk  categories  are  defined  and  examples 
are  provided  in  the  LOI. 

The  management  and  organizational  poli¬ 
cies  for  TRACE-P  are  also  similar  to  TRACE. 

The  PM  is  responsible  for  providing  the 
TRACE-P  estimate  in  conjunction  with  the 
Baseline  Cost  Estimate.  Upon  approval, 

TRACE-P  becomes  the  program  estimate  and  is 
submitted  to  DA  to  be  included  in  the  weapon 
system  procurement  line  item.  The  difference 
between  the  TRACE-P  and  BCE  will  be  identi¬ 
fied  as  the  Risk  Cost  or  TRACE-P  funds  which 
will  be  held  by  the  DCSRDA  and  released  to 
the  PM  only  upon  approval  by  DCSRDA  based  on 
a  wr-'tten,  validated  request  with  justifica¬ 
tion. 

The  following  methodologies  have  evolved 
for  TRACE-P: 

1 .  Investment  Phase  Management  Reserve 
(IPMR)  was  the  first  known  analysis  to 
attempt  to  quantify  the  cost  of  production 
risks  during  the  transition  period.  Devel¬ 
oped  for  the  Remotely  Piloted  Vehicle  program, 
IPMR  defines  uncertainty  elements  or  cate¬ 
gories  that  are  separate  from  the  BCE  struc¬ 
ture.  The  link  between  the  risk  elements  and 
the  BCE  is  the  Design  to  Unit  Production  Cost 
(DTUPC)  value.  The  impact  of  the  risk  ele¬ 
ments  on  the  DTUPC  is  estimated  as  a  triangu¬ 
lar  distribution  based  on  experts'  assessment 
of  the  nature  of  the  risk  and  its  cost  Impact. 
Then,  the  distributions  are  convolved  using 
Venture  Evaluation  and  Review  Technique 
(VERT)  which  produces  a  single  distribution 
for  cost  based  on  Monte  Carlo  techniques. 

The  uncertainty  elements  in  IPMR  are  labeled 
Production,  Performance,  Sizing,  Technology, 
Resources,  Management/Control,  Higher  Manage¬ 
ment  and  Dther. 

2.  Employing  a  similar  approach,  APRO 
and  ALMC  developed  an  alternate  methodology 
called  Production  Risk  Assessing  Methodology 
(PRAM)  based  on  a  survey  of  10  PM  offices,  3 
Defense  Contractors  and  several  DOD  experts. 
Linked  directly  to  the  BCE  are  11  risk  cate¬ 
gories  based  on  an  analysis  and  synthesis  of 
problems  cited  in  historical  systems  during 
the  transition  period.  The  11  risk  categories 
were  further  grouped  into  3  major  groupings 

as  shown  in  Figure  6.  For  each  risk  category 
an  assessment  is  made  on  the  possible  impacts 
on  the  affected  portions  of  the  BCE  as  a  cost 
distribution  such  as  triangular,  uniform,  nor¬ 
mal  or  other  distributions.  The  pertinent  BCE 
value  is  then  subtracted  from  each  distribu¬ 
tion  to  form  a  distribution  of  risk  costs  which 
can  then  be  combined,  assuming  independence, 
by  Monte  Carlo-based  techniques  such  as  VERT 
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or  by  application  of  the  Central  Limit  Theo¬ 
rem. 


riant  t  .  a  miik.  ruouenu  «:«  ■ 


3.  TRACE- P  methodology  outlined  in  the 
6  October  1982  LOI  was  developed  as  a  modifi¬ 
cation  to  PRAM.  Two  risk  categories  were 
eliminated,  one  redefined  and  the  hierarchy  or 
structure  eliminated.  The  estimation  of  cost 
At  distributions  as  specified  in  the  LOI  calls 
for  the  direct  estimation  of  risk  cost  or 
delta  costing,  although  the  BCE  value  is  to  be 
identified.  The  approved  risk  categories  for 
TRACE-P  are  defined  and  examples  provided  as 
Threat  Uncertainty,  Management,  Materials/ 
Purchased  Parts,  Facilities/Equipment,  Labor, 
Design  Changes,  Producibility  and  Performance. 
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4.  An  effort  to  develop  a  network-based 
TRACE-P  methodology  is  in  progress  at  the 
U.  S.  Army  Communication  and  Electronics  Com¬ 
mand  at  Fort  Monmouth,  NJ.  This  model  is  sim¬ 
ilar  to  the  Probabilistic  Network  Modelling 
technique  used  for  TRACE. 

Since  TRACE-P  is  so  recent,  little  exper¬ 
ience  and  data  would  be  available  to  evaluate 
the  effectiveness  of  the  program.  TRACE-P 
methodology  was  tested  by  reconstructing  an 
historical  system  BCE,  applying  TRACE-P  to  the 
pre-production  estimate  and  comparing  the  re¬ 
vised  estimate  to  the  actual  production  costs 
of  the  system  (M735  ammo).  The  revised  esti¬ 
mate  was  extremely  close  to  the  total  actual 
cost  and  also  matched  the  yearly  breakout  of 
costs  very  well.  However,  it  was  not  possible 
to  track  whether  or  not  the  actual  cost  in¬ 
creases  matched  the  risk  category  predictions 
because  of  data  limitations  on  the  actual 
costs.  Also,  the  experts  who  made  the  TRACE-P 
estimate  were  familiar  with  the  program  and 
knew  what  the  actual  problems  were,  facilita-. 
ting  the  correlation  of  revised  estimates  to 
actuals. 


because  the  TPACE-P  methodology  is  rela¬ 
tively  straightforward  and  easy  to  apply, 
many  recent  systems  have  submitted  TRACE-P  as 
part  of  the  program  estimate  for  the  POM 
process.  Time  will  tell  if  the  estimates  are 
accurate  and  the  methodology  sound.  Regard¬ 
less,  TRACE-P  will  increase  the  estimates  of 
early  production  costs  and  therefore  result 
in  fewer  cost  overruns  and  improved  manage¬ 
ment  of  the  acquisition  of  systems. 

COMPARISON  OF  TR4CE  AND  TRACE-P 


As  tna  name  indicates,  TRACE-P  is  a  nat¬ 
ural  extension  of  the  TRACE  program  from  Re¬ 
search  and  Development  into  the  initial 
phases  of  production.  Since  there  is  a 
carry-over  effect  of  technological  risk  in 
the  transition  phase,  the  TRACE-P  initiatives 
have  been  designed  and  patterned  a*ter  the 
earlier  TRACE  program.  Essentially,  some  of 
the  same  risks  associated  with  R&D  remain 
and  some  new  risks  develop  whenever  a  new 
product  is  produced  for  the  first  time  in  a 
production  environment,  rather  than  under 
prototype  conditions. 

Thus  there  is  a  great  deal  of  similarity 
between  TRACE  and  TRACE-P  in  the  policies  and 
implementation  guidance.  Both  programs  are 
implemented  through  a  LOI  as  opposed  to  a 
regulation.  Each  program  has  a  defined  scope 
of  uncertainties,  i.e.,  types  of  unc°rtain- 
ties  that  are  included  or  excluded.  In  fact, 
many  of  the  exclusions  are  conmon  to  both. 

Each  program  is  administered  the  same  way. 

The  PM  develops  both  estimates,  HQ  DARCOM  ap¬ 
proves  the  estimates  and  validates  the  justi¬ 
fication  for  release  of  funds.  DCSRDA  holds 
the  risk  funds,  manages  them  and  has  approval 
authority  for  their  release  to  the  PM.  Above 
the  DA  level,  the  TRACE  and  the  TRACE-P  be¬ 
come  part  of  the  weapon  system  line  item. 

DOD  and  Congress  see  a  single  unified  esti¬ 
mate  that  contains  the  sum  of  the  BCE  and  the 
risk  cost. 

In  addition,  there  is  a  great  deal  of 
similarity  between  the  analytical  techniques 
used  in  TRACE  and  TRACE-P.  Both  estimates 
rely  heavily  on  expert  judgment  and  the  use 
of  probability  theory  to  quantify  risks.  The 
early  TRACE  methodology,  application  of  risk 
factors  to  Work  Breakdown  Structure  elements, 
is  comparable  in  rigor  to  the  TRACE-P  method¬ 
ology  (neither  is  complex,  data  intensive  and 
time  consuming).  Both  estimates  suffer  from 
analytical  problems  steaming  from  the  inter¬ 
action  of  risks  or  interdependence  which  is 
difficult  to  quantify. 

The  differences  between  TRACE  and  TRACE-P 
are  more  obscure,  apart  from  the  differences 
associated  with  the  different  funding  appro¬ 
priations  involved.  Theoretically,  TRACE-P 
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should  be  a  smaller  percentage  of  the  procure¬ 
ment  program  than  TRACE  is  for  the  RDTE  pro¬ 
gram  because  it  only  applies  for  the  first 
three  years  and  because  technological  risk 
should  be  a  less  critical  factor  in  produc¬ 
tion.  On  the  ether  hard.  other  forms  of  risk 
besides  technical  barriers  may  tend  to  domi¬ 
nate  in  the  production  transition.  These 
risks,  identified  in  the  TRACE-P  LOI,  are  gen¬ 
erally  related  to  the  disruptions  to  the  pro¬ 
duction  process  and  planning  caused  by  a 
broader  scope  of  uncertainties  than  those  en¬ 
countered  in  R&D.  The  cumulative  effect  of 
the  many  small  nagging  problems  associated 
with  initial  production  that  frustrate  pro¬ 
duction  management  is  the  justification  for 
the  need  to  budget  funds  for  the  timely  res¬ 
olution  of  some  of  these  problems.  It  should 
be  noted,  however,  that  TRACE-P,  like  TRACE, 
does  not  cure  the  illness;  it  only  relieves 
some  of  the  symptoms.  Some  production  risks 
cannot  always  be  resolved  by  an  infusion  of 
additional  funds  in  a  timely  manner.  A  smooth 
transition  reouires  diligent  management  and 
planning,  beginning  in  the  early  stages  of  R&D 
regardless  of  the  availability  of  TRACE-P 
funds. 

PROGNOSTICATIONS 


With  the  knowledge  that  the  subject  is 
controversial  and  subject  to  widely  divergent 
opinions  within  and  among  the  servides,  a  pro¬ 
jection  of  the  future  of  TRACE-P  will  be  made 
based  on  the  Army  experience  with  TRACE.  Pre¬ 
dictions  (based  on  the  premise  that,  because 
the  programs  are  so  similar,  the  experiences 
of  TRACE  will  also  apply  to  TRACE-P)  are: 

1.  Acceptance  -  The  Army  will  implement 
TRACE-P,  slowly  at  first,  then  routinely  for 
major  systems.  Project  Managers  will  be  gen¬ 
erally  supportive  cf  the  program.  The  Navy 
and  the  Air  Force  will  also  fund  for  initial 
production  risks,  but  will  use  different 
models  and  procedures  to  do  so. 

2.  Improved  Methodologies  -  Alternate, 
more  sophisticated  methodologies  will  <i°- 
veloped  to  overcome  resistance  from  "budget- 
cutters"  and  "optimists."  These  models  will 
nonetheless  be  limited  by  the  nature  of  the 
subjective  data  inputs.  A  useable  risk  data 
base  will  never  be  developed  to  enable  ana¬ 
lysts  to  rely  less  on  expert  judgment. 

3.  Results  -  Programs  will  continue  to 
experience  problems  and  cost  overruns  during 
the  transition  from  R&D  into  production.  How¬ 
ever,  the  track  record  will  be  improved.  The 
rate  of  overrun  which  had  averaged  over  35%  in 
constant  dollars  during  this  phase  will  be  re¬ 
duced  to  less  than  10%  because  of  TRACE-P  and 
other  acquisition  reforms. 

4.  Criticism  -  There  will  continue  to  be 


critics  who  believe  that  risk  funds  will  be 
"self-fulfilling  prophesies,"  slush  funds  to 
cover  management  errors,  or  analytical 
empire-building  schemes.  They  will  be  sup¬ 
ported  by  the  inconsistency  of  programs  to 
not  use  their  risk  funds  (violating  the 
bu/50  principle).  This  criticism  will  result 
in  tighter  management  of  TRACE  and  TRACE-P 
reserve  funds,  denial  of  release  requests 
when  justification  is  inadequate  and  rewards 
to  PM's  who  do  not  use  all  of  the  risk  funds. 

5.  Expansion  -  In  the  long  term,  there 
will  be  other  applications  of  the  concept  of 
budgeting  for  risk.  Potential  areas  are 
initial  spare  parts,  military  construction 
projects,  procurement  of  foreign  systems, 
etc. 

In  summary,  the  Army  is  aggressively 
pursuing  the  policies  of  budgeting  for  risk. 

It  has  done  so  for  R&D  with  the  TRACE  program 
since  budget  year  1976  and  has  begun  a  follow- 
on  program,  TRACE-P,  to  budget  for  risk  in 
initial  production  in  FY  82.  The  two  programs 
are  similar  in  concept,  policy  and  methodol¬ 
ogy.  Thus,  for  the  Army,  history  does  seem 
to  be  repeated  in  the  TRACE-P  program. 
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ABSTRACT 


This  paper  reviews  formal  efforts  used  to 
identify  and  measure  risk  in  the  Remotely 
Piloted  Vehicle /tREV)'  Program.  Formal  means 
are  contrasted  with  informal  efforts  which 
also  surface  risk  areas,  yet  are  designed  for 
other  purposes.  Discussions  follow  to  empha¬ 
size  specific  actions  taken  by  the  Project 
Manager'TPM)  tc  minimize  identified  risk. 

PART  ONE  \ 

During  the  course  of  the  RPV  Program,  various 
attempts  have  been  made  to  quantify  the  risk 
associated  with  the  Full  Scale  Engineering 
Development  (FSED)  Phase  and  the  Investment 
Phase.  In  mid-year  1979,  a  Total  Risk  Assess¬ 
ing  Cost  Estimate  (TRACE)  was  estimated  that 
equalled  15%  of  the  total  RPV  FSED  program 
cost.  This  estimate  was  based  upon  a  com¬ 
parison  with  the  Advanced  Attack  Helicopter 
(AAH)  and  Target  Acquisition  Designation 
System  (TADS)  Piloted  Night  Vision  System 
(PNVS)  weapon  systems  and  should  be  assessed 
in  light  of  the  RPV  program's  lack  of  maturity 
at  that  time. 

In  June  of  1981,  when  the  Army  was  considering 
the  value  of  a  TRACE  concept  for  the  Investment 
Phase  of  the  Life  Cycle  Cost  Estimate  (LCCE) , 
a  report  was  submitted  which  described  the 
analysis  performed  in  calculating  a  risk 
assessment  of  the  RPV's  Investment  Phase. 

This  "Management  Reserve"  as  it  was  called, 
categorized  risk  into  the  following  categories: 


Production 

Performance 

Sizing 

Technology 

Resources 

Politics 

Other 


This  report  identified  the  Air  Vehicle,  Mission 
Payload,  and  Modular  Integrated  Communications 
Navigation  System  (MICNS) ,  as  having  high 
production  risk.  It  assigned  MICNS  as  high 
risk  in  terms  of  performance,  sizing,  and 
technology.  The  Mission  Payload  also  ranked 
high  in  sizing.  Politics  was  a  significant 
risk  area  for  the  recovery  subsystem;  probably 
because  the  recovery  system  is  supplied  by  a 
foreign  company.  Overall,  the  analysis 
recommended  Getting  aside  18  percent  of  the 


investment  cost  for  Management  Reserve. 

Department  of  the  Army  (DA)  formed  a  team  in 
August  1982  to  discuss  preliminary  issues  then 
being  considered  by  a  formal  risk  team,  whose 
purpose  was  to  recommend  a  cost  position  to  the 
Army  System  Acquisition  Review  Council  (ASARC) 
Chairman.  The  items  that  posed  some  degree 
of  risk  included: 

.  Changes  in  the  Organizational  and 
Operational  (0&0)  concept. 

.  Environmental  protection  requirements 
(nuclear  hardening,  etc.). 

.  Composite  optic's  producibility . 

.  Semiconductor  availability. 

.  Transition  from  FSED  to  production. 

.  Production  facilities  move  from 
California  to  Austin,  Texas. 

.  Funds  stability. 

A  short  time  after  the  DA  Risk  Team  completed 
their  evaluation,  a  Total  Risk  Assessing  Cost 
Estimate-Production  (TRACE-P)  ad  hoc  team 
convened  to  incorporate  the  findings  into  an 
updated  TRACE-P  estimate.  The  DA  Risk  Team 
assigned  high  risk  to  the  Data  Link;  the  move 
to  Austin;  Lockheed  Missile  and  Space  Company's 
(LMSC's)  production  planning  and  their 
technical  ability  to  integrate  the  system; 
LMSC's  ability  to  provide  the  necessary 
composite  optics  and  software  independent 
verification  and  validation;  and  also  the  need 
to  obtain  a  larger  Ground  Control  System  (GCS) 
computer.  The  team,  using  a  statistical 
program  to  define  cumulative  distributions, 
placed  the  risk  team's  concerns  into  the 
following  categories: 

Threat  Uncertainty 
Management  (i.e.,  Funding) 
Materials/Purchased  Parts 
Facilities /Equipment 
Labor 

Design  Changes 
Producibility 

The  highest  risk,  in  terms  of  expected  cost, 
was  under  the  Facilities/Equipment  category. 
The  basic  cause  of  this  high  potential  cost 
was  the  possible  requirement  to  open  the 
acquisition  of  the  Air  Data  Terminal  to  compe¬ 
titive  bidoing  and,  also,  the  possibility  of 
having  different  subcontractors  work  in  the 
production  phase.  The  composite  optics  was 
responsible  for  the  high  risk  assigned  to 
Materials/Purchased  Parts  and  to  a  lesser 
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extent,  the  risk  assigned  to  Producibility . 

The  move  to  Austin  generated  risk  associated 
with  Labor.  The  concern  here  was  the  possible 
growth  of  burdened  labor  rates  and  a  possible 
facilities  contract  with  the  Government. 

Another  effort  was  being  undertaken  during  the 
same  timeframe  to  assess  the  risk  associated 
with  the  FSED  Program.  The  resulting  document 
represented  a  Decision  Risk  Analysis  (DRA)  for 
the  RPV  PMO.  Its  purpose  was  to  determine  the 
performance,  cost,  and  schedule  risks  in  the 
Research  and  Development  (R&D)  Phase. 

Out  of  forty-two  parameters  identified  repre¬ 
senting  performance,  six  were  assigned  high 
risk: 

.  Video  Downlink  (MICNS) 

.  Displacement 

.  Air  Vehicle  Design  Gross  Weight 
.  Forward  Looking  Infrared  (FLIR)  Weight 
.  Mission  Payload  Weight 
.  Electrical  Optical  Augmentation  Test 
Program  Sets 

Cost  risks  were  separated  into  three  types: 

.  Cost  Uncertainty 
.  Fixed  Cost  Growth 
.  Variable  Cost  Growth 

Significant  risk  items  influencing  the  schedule 
were: 

.  Number  of  Test  Program  Sets  (TPSs) 

.  Electrical  Optical  Augmentation 
.  Mission  Payload  Redesign 
.  MICNS  Slip  (Redesign  and  Integration) 

.  TPS  Development 
.  FLIR 

The  DRA  indicated  eight  percent  of  FSED  costs 
for  the  TRACE  deferral.  In  February  1983,  an 
update  to  the  DRA  was  accomplished  that  con¬ 
sidered  only  a  revision  to  the  RPV  schedule. 
Performance  and  cost  were  assumed  unchanged. 

PART  TWO 

Drawing  upon  the  results  of  the  studies  and 
reports  examined  in  Part  One,  the  Project 
Manager  (PM)  recognizes  the  high,  moderate 
and  low  areas  of  risk  in  his  program.  He 
evaluates  the  status  of  his  program,  determines 
whether  adequate  funding  is  available,  and 
takes  appropriate  steps  to  reduce  his  program 
risk  level. 

Risk  can  be  identified  by  informal  and  formal 
means.  The  informal  means  are  these  activities 
that  surface  risk  areas,  but  are  not  designed 
for  that  particular  purpose.  Some  examples 
are : 


.  The  PM  and  the  prime  contractor  meet 
regularly  on  a  monthly  basis  to  discuss 
current  problems  and  other  issues  relevant 
to  the  program. 

.  An  Informal  Technical  Review  (ITR) 
which  is  a  command  unique  concept  where  the 
PM  sends  out  a  team  to  the  subcontractors  in 
order  to  initiate  schedules,  monitor  progress, 
and  assist  in  problem  areas,  may  also  surface 
risk. 

.  Factfinding  teams  can  also  surface 
areas  although  this  is  not  its  intended 
purpose. 

Formal  means,  on  the  other  hand,  are  those 
studies,  teams,  and  other  efforts  devised 
specifically  to  identify  risk  areas. 

Examples  of  the  observable  actions,  formal 
and  informal,  taken  by  the  RPV  PM  to  manage 
his  program's  risk  as  brought  out  in  Part  One 
will  be  examined. 

Most  of  the  actions  taken  by  the  PM  to  mini¬ 
mize  risk  involves  the  relationship  that  he 
has  developed  with  the  prime  contractor.  If 
the  PM  has  earned  the  respect  of  the  con¬ 
tractor;  if  he  has  convinced  the  contractor 
that  he,  the  PM,  represents  the  Army  and  wants 
what  is  best  for  the  Army;  if  he  has  been  fair 
and  capable  in  the  past,  then  the  effort 
exerted  by  the  prime  contractor  should  be 
satisfactory  in  reducing  risk,  as  long  as  the 
risk  areas  remain  monitored  by  the  PM. 

However,  there  are  times  when  the  PM  has 
accomplished  all  of  the  above  and  the  prime's 
efforts  still  do  not  reduce  the  level  of  risk 
to  the  PM's  satisfa  iion.  Consequently,  there 
are  necessary  actions  that  the  PM  must  take  to 
resolve  intolerable  levels  of  risk  in  his 
program. 

Soon  after  the  contractor  had  begun  work  on 
the  RPV  contract,  the  PM  directed  the  effort 
that  culminated  in  the  June  81  report.  The 
findings  indicated  to  the  PM  that  MICNS  was 
high  risk  in  terns  of  getting  it  to  operate 
satisfactorily  at  the  dimensions  required. 

In  fact,  size  was  perhaps  the  PM's  greatest 
concern  at  that  time  for  both  MICNS  and  the 
Mission  Payload.  In  addition,  due  to  the 
business  conditions  at  the  responsible  sub¬ 
contractor,  concern  was  expressed  about  the 
level  of  risk  in  the  Air  Vehicle.  Such 
conditions  were  unfavorable  labor  practices, 
lack  of  production  rapacity,  and  poor 
management. 

To  alleviate  these  risks,  the  PM  took  three 
observable  actions  that  have  paid  off: 

.  The  subcontractor  for  the  airframe  was 
switched  from  the  prime  contractor's  sister 
company  Georgia  Lockheed  (GELAC)  to  Hltco. 

.  The  program  was  stretched  out  from 
43  to  52  months. 

.  Dual  second  tier  subcontractors  were 


established  to  develop  and  integrate  mission 
payload  composite  optics. 

These  actions  significantly  reduced  the  risks 
documented  in  the  Management  Reserve  Study. 

Approximately  a  year  later,  the  program  was 
again  assessed  for  the  purpose  of  reducing 
risk.  The  DA  Risk  Team  and  the  TRACE-P  Team, 
who  conducted  a  follow-up  study,  identified 
theit  areas  of  concern  for  that  period  of 
time.  These  concerns,  which  have  been  listed 
above,  shed  light  on  some  of  the  special 
actions  taken  by  the  PM  which  have  greatly 
alleviated  the  level  of  risk.  For  example, 
the  DA  team  noted  that  funding  perturbations, 
which  marked  the  program  in  early  FY82,  had 
considerable  risks  associated  with  it  and 
questioned  how  the  PM  would  minimize  any 
future  funding  swings.  One  technique  used 
by  the  RPV  PMO  is  a  heavy  reliance  on  the 
Baseline  Cost  Estimate  (BCE)  in  executing  the 
budgetary  process.  By  utilizing  the  BCE  in 
this  way,  even  before  the  Carlucci  Initia¬ 
tives  required  it,  the  PM  recognized  the  im¬ 
plications  of  budget  swings  on  his  program 
and,  thus,  could  take  appropriate  actions. 

With  regard  to  the  other  risk  areas,  the  PM 
set  up  TIGER  teams  who  were  sent  to  the  sub¬ 
contractors  to  initiate  schedules,  monitor 
progress,  and  assist  in  problem  areas.  The 
PM  also  set  up  Informal  Technical  Reviews 
(ITRs)  which  keeps  the  PMO  and  the  contractor 
in  close  communication.  As  a  further  example 
of  PM  action,  he  is  smoothing  the  transition 
from  FSED  to  Investment  by  establishing  an 
early  operational  capability  program  which 
provides  a  RPV  section  to  a  US  Division. 

Also,  through  a  Producibility  Enhancement 
Initiative  (PEI) ,  he  Is  providing  for  units 
to  be  built  with  hard  tooling  during  the 
transition  phase  between  R&D  and  Production. 

A  final  example  of  actions  taken,  relative 
the  DA  Risk  Team,  is  the  phasing  out  of 
engineers  from  Sunnyvale ,  California  to 
Austin,  Texas.  Originally,  It  was  planned 
to  move  all  of  the  engineers  to  Austin  by  a 
certain  date;  however,  to  minimize  disruption, 
only  those  whose  work  is  finished  at  Sunnyvale 
are  allowed  to  move  to  Austin. 

Next,  the  TRACE-P  Team  applied  a  statistical 
analysis  of  the  risk  areas  identified  by  the 
DA  team.  A  point  known  to  the  DA  team  and 
emphasized  firmly  by  the  TRACE-P  Team  con¬ 
cerned  risk  associated  with  the  Test  Program 
Sets  (TPSs) .  Because  newly  developed  weapon 
systems  are  often  highly  complex  in  terms  of 
their  construction  and  capability,  and  rely 
heavily  on  software  either  in  operation  or 
testing  for  faults,  the  strategy  taken  by  the 
RPV  PM  is  to  develop  about  15Z  of  the  TPSs 
during  FSED  and  to  construct  the  remaining 
sets  after  FSED  is  completed.  Having 
workable  computer  programs  developed  in  this 
way  should  minimize  program  risk  considerably 


in  terms  of  reducing  computer  reprogramming, 
often  necessitated  by  engineering  changes  in 
the  hardware  being  developed  during  FSED. 

The  last  risk  study  under  consideration  here 
is  a  Decision  Risk  Analysis  (DRA) .  The  data 
base  for  this  effort  was  a  questionnaire  on 
which  the  responsible  PMO  engineer  described 
the  risk  level  for  one  of  42  parameters  and 
the  plan  formulated  to  manage  the  associated 
risks.  A  review  of  the  responses  underlines 
the  PM's  obvious  concern  for  risk  items  and 
the  reduction  in  risk  resulting  from  actions 
which  the  PM  has  taken.  For  instance,  the  PM 
performed  tradeoff  studies  to  evaluate  Organi¬ 
zational  and  Operational  (0&0)  concepts,  which 
impact  displacement,  and  simultaneously  allows 
the  PM  to  plan  for  potential  scenarios.  Test 
Program  Sets  (TPSs)  remain  a  concern  and  are 
continually  monitored  by  management  and  tech¬ 
nical  personnel,  as  noted  above.  In  addition, 
programs  written  for  TADS/PNVS  are  being 
studied  for  application  to  the  RPV  program's 
test  sets. 

CTTVOjCApV 

u  uiuuiiv  i. 

The  key  to  managing  the  RPV  Program's  inherent 
risk  is  early  identification  of  the  problem 
areas.  The  PM  augmented  the  usual  means  of 
assessing  risks,  e.g.,  Initial  Production 
Readiness  Reviews  (IPRRs) ,  factfinding, 
standard  reviews,  and  monthly  status  meetings, 
with  studies  designed  specifically  for  Iden¬ 
tifying  and,  to  some  extent,  quantifying 
program  risk.  Like  so  many  other  state-of-the- 
art  weapon  system  programs,  the  areas  of  risk 
change  and  the  levels  of  risk  tend  to  diminish 
as  the  program  matures.  The  RPV,  however, 
insures  that  Its  program  endures  only  minimum 
risk  by  early  identification  and  good  manage¬ 
ment  practices.  Those  risk  areas  identified 
in  the  recent  DRA  are  indeed  different  from 
those  Identified  in  June  1981.  Moreover, 
the  level  of  risk  fcr  each  item  is  considered 
to  be  substantially  reduced. 
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ABSTRACT 

/  ' 

Thia  paper  presents  a  brief  description  of  a 
computer  program  which  is  available  to  perform 
calculations  needed  in  a  risk  analysis.  The 
model  discussed  in  this  paper  is  one  which 
allows  a  user  to  estimate  the  risk  associated 
with  any  number  of  variables  and  to  display  the 
distribution  of  any  arithmetic  (addition, 
subtraction,  and  (or)  multiplication) 
combination  of  the  variables. 

The  mode  of  operation  is  designed  to  be  similar 
to  a  calculator.  Rather  than  entering  in  a 
single  number,  the  user  must  supply  a  low,  most 
likely  and  high  for  each  variable.  Variables 
can  be  added,  subtracted,  or  multiplied; 
intermediate  calculations  can  be  stored;  and 
the  distribution  of  the  total  can  be  displayed 


INTRODUCTION 


relationship  between  the  different  variables 
involved  to  be  defined  in  advance.  Either  of 
these  requirements  limit  the  flexibility  of  the 
software  and  hence  applications. 

The  computer  program  discussed  in  this  paper 
was  designed  such  that 

-no  formal  training  is  required  on  the 
modeling  technique  employed, 

-no  mathematical  equations  need  to  be 
defined, 

-the  operation  is  similar  to  a 

calculator,  and 

-the  results  can  be  displayed 

immediately . 

The  program  can  be  used  in  numerous  areas  of 
applications  including,  but  not  limited  to, 
quality  control,  scheduling,  inventory  control, 
and  cost  analysis. 


Many  point  estimates  of  totals  (e.g.,  project 
completion  time,  cost  of  weapon  systems,  etc.) 
involve  simple  addition,  subtraction  and 
multiplication  which  can  be  performed  on  a  hand 
calculator.  Commonly,  point  estimates  are  made 
for  each  subcomponent  and  totals  are 
calculated.  Problems  arise  when  there  is 
uncertainty  about  the  value  of  the  individual 
component  because  of  lack  of  information  or  the 
inability  to  accurately  forecast  the 
subcomponent.  The  affect  of  this  uncertainty 
on  the  total  is  captured  by  risk  analysis. 

To  determine  the  amount  of  uncertainty  4n  a 
total,  an  analyst  would  not  only  estimate  the 
most  likely  value  for  each  subcomponent  but 
also  estimate  the  probability  of  different 
outcomes  for  each  subcomponent.  From  this 
information,  probabilities  of  different  totals 
could  be  calculated.  This  is  exactly  what  is 
done  in  statistical  risk  analysis.  The  only 
difference  is  in  the  way  that  an  analyst 
supplies  information  about  the  uncertainty  in 
subcomponents  and  how  this  infc  mation  is  used 
to  make  probability  statements.  The  complexity 
of  the  calculations  required  make  the  use  of  a 
computer  almost  a  necessity. 


Interactive  Model 

The  computer  program  being  discussed,  accepts 
three  estimates  for  each  subcomponent  and 
allows  the  user  to  combine  the  subcomponents 
using  any  sequence  of  addition,  subtraction  and 
multiplication.  The  three  points  required  are  a 
most  likely  value  (i.e.,  the  value  with  the 
highest  probability),  a  low  value  (i.e.,  a 
value  which  will  be  underrun  only  12  of  the 
time),  and  a  high  value  (i.e.,  a  value  which 
will  be  overrun  only  12  of  the  time).  When  the 
user  has  combined  the  subcomponents  in  a 
meaningful  fashion  the  probability  of  different 
totals  can  be  requested. 

The  user  has  the  following  commands  available: 
-ADD  for  addition 
-SUB  for  subtraction 
-MUL  for  multiplication 

-STO  for  storing  intermediate  calculations 
-RCL  for  recalling  stored  information 
-INP  for  inputing  the  thr.ee  point  estimates 
-CON  for  displaying  the  contents  of  storage 
-DIS  for  displaying  probabilities 
-BYE  to  end  program. 


Most  computer  programs  available  to  assist  an 
analyst  in  performing  calculations  required  in 
a  risk  analysis  are  designed  with  a  particular 
area  of  application  in  mind  or  require  a 


Note,  there  is  no  hierarchy  of  operation  (i.e., 
1  added  to  2  and  then  multiplied  by  3  would  be 
9  not  7). 
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EXAMPLE 


In  order  to  illustrate  the  interactive  risk 
program,  a  simple  example  will'  be  used.  The 
problem  is  to  add  two  variables  together  and 
then  to  multiply  by  a  third  variable.  The  data 
for  the  three  variables  is  shown  below. 


Variable 

Low 

Most  Likely 

High 

1 

300  ' 

400 

600 

2 

100 

300 

900 

3 

.1 

.3 

.5 

INTERACTIVE  RISK  ANALYSIS  EXAMPLE 

COMMAND  (mul,  add,  sub,  dis,  inp,  sto,  rcl,  con,  bye) 

?  inp 

Low,  Most  likely,  and  High  ?  300 , 400 , 600 

COMMAND  (mul,  add,  sub,  dis,  inp,  sto,  rcl,  con,  bye) 

?  add 

COMMAND  (inp,  rcl)?  inp 

Low,  Most  likely,  and  High  ?  100,300,900 

COMMAND  (mul,  add,  sub,  dis,  inp,  sto,  rcl,  con,  bye) 

?  sto 

Enter  numeric  storage  location?  21 

COMMAND  (mul,  add,  sub,  dis,  inp,  sto,  rcl,  con,  bye) 

?  inp 

Low,  Most  likely,  and  High  ?  .1,.3,.5 

COMMAND  (mul,  add,  sub,  dis,  inp,  sto,  rcl,  con,  bye) 

?  mul 

COMMAND  (inp,  rcl)?  rcl 

Enter  Numeric  Storage  Location?  21 

COMMAND  (mul,  add,  sub,  dis,  inp,  sto,  rcl,  con,  bye) 

?  dis 

PROBABILITY  TOTAL  Most  likely  value=  2 . 0194662D+02 

OF  EXCEEDING  VALUE  Mean  =  2 . 4066549D+02 

0.01  5 . 00706  85D+02 
0.05  4.1430178D+02 
0.10  3 . 7006158D+02 
0.15  3 . 4109697D+02 
0.20  3 . 1865877D+02 
0.25  2. 9986448D+02 
0.30  2 . 8337811D+02 
0.35  2.6845837D+02 
0.40  2 . 5464213D+02 
0.45  2 . 4161250D+02 
0.50  2 . 2913463D+02 
0.55  2.1702022D+02 
0.60  2 . 0510498D+02 
0.65  1 . 9323140D+02 
0.70  1 . 8123174D+02 
0.75  1 . 6890555D+02 
0.80  1 . 5598177D+02 
0.85  1.4203856D+02 
0.90  1 . 26  28802D+02 
0.95  1 . 0673596D+02 
0.99  8.1«82755D+01 


The  actual  input  and  output  is  shown  below.  The 
use  of  the  STO  command  was  not  necessary  but 
was  used  to  demonstrate  how  intermediate 
calculations  can  be  stored.  The  underlined 
portion  is  the  user  input. 
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The  program  will  allow  for  up  to  100 
distributions  of  intermediate  calculations  to 
be  stored.  The  number  of  inputs  and 
calculations  is  unlimited. 


SUMMARY 

An  interactive  risk  analysis  program,  which  is 
not  tied  to  a  particular  application  and  does 
not  require  modeling  knowledge  of  the  user,  has 
been  described  in  this  paper.  The  ability  to 
perform  risk  calculations  in  a  manner  similar 
to  using  a  calculator  makes  the  program  easy  to 
use  and  adds  flexibility. 

The  program  is  currently  available  on  two 
brands  of  micro  computers  and  will  be  on  COPPER 
IMPACT  in  the  near  future.  For  more 
information  concerning  the  program  or  its  usage 
contact  the  author  or 

Business  Research  Management  Center 
AFBRMC/RDCB 

Wright-Patterson  AFB,  Ohio  45433 

The  development  of  this  program  was  supported 
by  the  Air  Force  Business  Research  Management 
Center. 
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BSTRACT 


-.This  paper  describes  a  microcomputer 
C—J  program  that  is  used  at  Mart  i-n 

Marietta  Denver  Aerospace'tMMDAT'  to 
perform  quantitative  risk  assessment 
and  to  assist  in  program  risk 
management.  Examples  are  presented  to 
illustrate  its  use  for  analyzing 
technical , schedule,  and  cost  risks. 

The  program  has  been  developed  to 
require  minimal  training  in  order  to 
provide  a  tool  that  program  managers 
can  and  will  use.  The  functional 
capabilities  of  the  program  for 
monitoring  and  managing  risk  data  are 
4  described  including  examples  showing 
user  interaction. 

I.  Introduction 

This  paper  describes  a 
microcomputer  hosted  risk  assessment 
program  that  is  designed  for  direct 
use  by  program  managers  or  systems 
engineering  leads  even  though  they  may 
have  no  previous  experience  using 
computers.  The  software  is  part  of  a 
risk  analysis  methodology  developed 
and  being  implemented  by  the  Systems 
Engineering  Department  of  MMDA.  The 
program  has  been  named  RAMP,  for  Risk 
k  Assessment  and  Management  Program.  It 
should  not  be  confused  with  a 
similarly  named  program  developed  by 
the  USAF  in  the  1970’ s. 

RAMP  provides  an  easily  used  tool 
to  maintain  program  risk  data  and  to 
analyze  program  technical,  cost,  and 
schedule  risks.  Using  RAMP,  managers 
may  interactively  compare  alternative 
program  strategies  proposed  to 
mitigate  risks  and  identify  effects  of 
potential  problems.  The  ability  to 
rapidly  evaluate  risks  for  program 
alternatives  is  a  desirable  capability 
in  today's  acquisition  environment.  A 
recently  completed  USAF  study 
indicates  that  in  the  1970’s  the 
major  sources  of  program  problems  are 


funding  instability,  external 
manaqement  impact,  and  technical 
complexity.  Each  of  these  effects 
will  modify  requirements  and  program 
hierarchies,  introducing  new 
uncertainties  thus  requiring 
development  of  new  program  strategies 
and  increasing  the  need  for 
quantitative  risk  assessment.  An 
interactive  risk  assessment  tool  can 
permit  managers  to  guickly  identify 
effects  of  such  potential  program 
perturbations. 

Another  attractive  feature  of  RAMP 
lies  in  the  security  of  program  data 
inherent  with  use  of  a  mi crocomputer 
for  sensitive  programs  or  for 
sensitive  managers. 

The  MMDA  risk  analysis  approach 
that  incorporates  RAMP  is  shown  in 
Figure  1.  A  fundamental  feature  is 
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the  continuing  assessment  of  all 
program  risks  from  preproposal 
activities  through  program  completion. 
Early  in  a  program,  risk  assessment 
prnvi Jes  inputs  for  deriving  cost  and 
schedule  estimates  that  include  risk 
induced  by  technical  uncertainties. 
Just  as  important,  the  risk  assessment 
methodology  helps  identify 
inconsistent  or  deficient 
requirements.  It  is  impossible  to 
make  estimates  for  technical  risk 
parameters  without  11  defined 
performance  requirements.  As  shown  in 
Figure  1,  the  customer  is  involved  in 
the  overall  risk  management  process 
since  risks  may  sometimes  have  to  be 
balanced  against  mission  goals  or  need 
of  a  weapon  system  to  counter  a 
threat.  Figure  2  shows  how  systems 
engineering  utilises  the  RAMP  data 
base  to  ensure  that  all  disciplines 
involved  in  program  performance, 
monitoring,  estimating,  and  planning 
use  consistent  and  timely  risk  data. 

Following  sections  describe  key 
features  of  RAMP  and  present  examples 
to  illustrate  its  application. 

II.  RAMP  Description 

RAMP  has  been  implemented  on  both 
IBM  PC  and  APPLE  11+  mi crocomputer s. 

It  has  also  been  recently  converted 
and  implemented  on  a  VAX  11/780 
computer  for  use  by  some  programs 
which  do  not  have  access  to 
mi crocomputer  systems.  Some  of  the 
calculation  algorithms  were  initially 


developed  on  an  HP-85  desk  top 
computer  and  were  modified  for  the  IBM 
PC. 

RAMP  provides  "manager — friendly" 
executive  control,  data  base 
management,  risk  calculation  using 
Monte  Carlo  simulation,  and  tabular 
and  plotted  output  display  of  risk 
effects  in  the  form  of  cumulative 
probability  functions  for  risk 
parameters.  It  does  not  incorporate 
network  models  of  programs.  Rather, 
it  is  intended  to  identify  a  range  o-f 
values  for  a  single  program  risk 
parameter  that  is  not  uniquely  defined 
because  of  some  program  uncertainty  or 
combination  of  uncertainties. 

Examples  of  risk  parameters  that  have 
been  assessed  for  programs  are 
required  lines  of  software  code, 
system  mass,  time  required  to  complete 
sets  of  tasks,  engineering  manpower 
and  associated  cost,  probability  of 
program  success  (or  overall  program 
risk),  and  relative  complexity  or 
technical  state  of  the  art  increase 
required  for  a  program. 

Following  are  paragraphs  which 
discuss  the  RAMP  executive,  data 
archi tecture.  and  data  manager. 
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Figure  2  MMDA  Risk  Methodology  Interfaces 
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II. 1  RAMP  Executive  Approach 

The  basic  objective  is  simplicity 
of  use,  both  with  respect  to 
minimicing  training  required  for  use 
and  eliminating  ambiguity  in  user 
prompts.  Experience  indicates  that 
managers  most  likely  will  not  use 
computer  tools  that  requi re  a  learning 
process  or  relearning  if  time  passes 
between  uses.  For  this  reason,  the 
RAMP  executive  software  is  based  on 
menu  selection  and  unambiguous  prompts 
requiring  either  YES/NO  answers,  user 
defined  file  names,  or  user  defined 
parameter  names.  All  user  inputs  are 
protected  against  inadvertent  input 
errors  or  reading  through  end-of-data. 
Preliminary  testing  has  shown  that 
RAMP  may  be  used  effectively  with  less 
than  10  minutes  training.  The  Main 
Menu  display  shown  in  Example  1 
permits  a  user  to  identify  and  select 
the  desired  risk  assessment  function. 


Select  Risk  Assessment  Function: 

1 - Enter  new  r i sk 

2- Modify  current  risk 

3- Delete  risk  element 
-•-Display  risk  data 
5-Create  risk  calculation  file 
S-Calculate  and  Plot  risk  effect 
7-Stop 

Function  choice  1 

Enter  Risk  DB  File  Name  ?  PAPERDB 


Example  1  RAMP  Main  Menu 


1 1.2  RAMP  Data  Management 

The  data  base  stores  those  data 
required  for  the  quantitative  risk 
assessment.  The  data  are  obtained  by 
a  cooperative  effort  including  systems 
engineers,  technical  leads,  cost 
estimators,  etc.  as  shown  previously 
in  Figure  2.  Data  relating  to 
rationale  for  the  risk  assessment, 
when  made  and  by  whom,  and  information 
relevant  to  risk  management  are 
presently  maintained  on  hardcopy  forms 
used  to  acquire  data.  Only  data 
required  for  the  RAMP  calculation  is 
entered  into  the  program  data  base. 


RAMP  Data  Architecture 

Each  RAMP  data  base  record 
contai ns : 

1)  a  task  identifier  (e.g.  WBS 
number ) ; 

2)  a  risk  type  identifier  where 

1  *  performance  risks 

2  *»  supportabi  1  ity  risks 

3  =  schedule  risks 

4  =  cost  risks; 

3)  a  risk  parameter  name; 

4)  a  probability  density  function 
type  where 

1  =  uniform 

2  =  triangular 

3  =  normal 

4  =  fixed  value; 

5-7)  values  corresponding  to  the 
density  function. 

All  risk  data  for  a  proqram  are 
stored  in  a  program  unique  data  base 
using  a  host  computer  random  access 
file.  when  calculation  of  a 
cumulative  function  is  desired  for  a 
risk  parameter,  a  calculation  file  is 
created.  The  calculation  file  is  a 
host  sequential  file  containing  the 
density  type  and  values  for  each 
record  of  the  specified  risk 
par ameter . 

RAMP  Data  Management 

The  first  5  Main  Menu  choices  are 
data  management  functions.  Choice  1 
is  used  to  create  a  new  data  base  or 
to  add  new  risk  records.  Choice  2 
permits  individual  risk  data  records 
to  be  modified  while  choice  3  permits 
deletion  of  a  risk  record  from  the 
data  base,  either  of  whicu  might 
happen  as  a  program  matures  and  risks 
are  closed  out.  A  typical  use  jf 
these  functions  would  result  i J  a 
manager  desired  to  tradeoff  rink 
resulting  from  alternative  approaches 
to  mitigate  risks.  A  second  use  is  to 
determine  the  effects  if  estimates  are 
believed  to  be  verly  pessimistic  or 
op t  i  mi  sti  c . 

Choice  4  is  used  to  display  risk 
data  records.  The  submenu  shown  in 
Example  2  allows  selective  display  of: 

1)  all  data  records! 

2)  all  records  of  a  single  WBS  or 
for  a  single  furction; 

3)  all  records  of  a  single  risk 
parameter ; 

4)  all  records  of  a  risk  type. 


83 


Runet  cri  c'.::ci  °  J 

Ent®r  Ri  ■  k  DS  "il®  N  =  »®  ?  PAPERR2 

Wcvld  V : u  r  i’-f®  A  Hard  Copy'3  N 
S®  ’  ®c*  Rigk'Dats  Pisclsv  Mod® 

-- Display  all  data 
2  -  E i s~ ’ ay  bv  WE? 

3-D-.5T-.iav  by  r-ak  tarameter 
-  -  D  :  jq  J  ;v  b  v  r  ’  rk  t  vp® 

5-R* turn  to  MAIM  MSS l’ 

•  D  :  5D 1  a-  Choica? 


Fane  t 1  «n  CH*1C*  *  5 

C*  rr«n*  SI  Kit**  I*  FAFBKSI  Vh  Thi*  SI* 

*  T 

Of  C-Ic.  *r  »• I toi 1 <»t l ••  Di.trib.tun  Fit*  T*  K*  Cr**«*4>T  OIAUi 

C.lcal.t.  CuwIiiih  Ktik  Di*trik.ti«n 
!*lt«  Kl#k  TY3* 

I -F*rf *r**ne • 

Mutirukil  It* 

**Ca*t 

tl.fc  F.f*..«.r  I.”  "TKO-KKK 
Ft  I*  n***4  01AUL  cr**t*4  • 


Example  4  Create  Calculation  File 


Example  2  RAMP  Data  Display  Menu 


RAMP  Risk  Calculation 


A  user  prompt  is  issued  at  the  start 
of  the  display  function  to  control 
display  for  a  printer  or  display 
monitor.  If  a  printer  is  not  used  the 
display  will  proceed  one  risk  record 
at  a  time.  A  record  display  and 
prompt  to  continue  is  shown  in  Example 
3.  When  the  selected  display  function 
is  completed  the  display  menu  is  again 
presented  for  further  display  or 
return  to  the  Main  Menu. 


RECORD  » 

WES  1 2 . 0 
fast  risk 


Paramet®r--D!RECT  LABOR 


Distribution  Type 
Lower  Limit 
Upper  Limit 
Most  Likely  Value 


Is  Triangular! 
4400 


5700 

4300 


2  ) 


To  Continue,  Type  Any  Kay,  Then  Return 
To  Stop, Just  Return13  E 

Example  3  Risk  Data  Display 


Choice  5  creates  a  sequential 
data  file  to  be  used  during  risk 
calculation.  The  file  is  created  by 
user  input  of  the  desired  risk 
parameter  as  shown  in  Example  4.  The 
data  base  is  searched  for  all  records 
of  the  parameter  and  the  parameter 
name,  probability  density  function 
type,  and  risk  parameter  data  are 
written  to  the  calculation  file. 
Several  calculation  files  can  be 
created  at  this  time  before  entry  to 
the  calculation  section  of  RAMP. 


RAMP  calculates  risk  effects 
using  the  probability  density  function 
of  each  risk  record.  At  present 
calculation  algorithms  are  included 
for  uniform,  normal,  and  triangular 
distributions.  An  algorithm  for  Beta 
distributions  could  easily  be  added  if 
desired.  The  increased  accuracy  in 
fitting  a  density  function  to  a  Beta 
distribution  is  probably  i nsi gni f i cant 
when  compared  to  the  uncertainty  in 
the  risk  data  estimates.  The  ability 
of  triangular  functions  to  incorporate 
skewed  estimates  is  considered 
sufficient  for  the  RAMP  analyses. 

When  the  calculation  section  is 
entered  the  user  is  prompted  as  shown 
in  Example  5  to  select  the  risk 
dependency  mode  that  best  reflects  the 
program  and  the  particular  parameter 
being  analyzed.  The  choice  will  be 
for  either  dependent  risks  or 
independent  risks.  The  variance  for 
dependent  risks  will  be  greater  than 
for  independent  risks.  In  reality, 
some  program  risks  will  be  independent 
and  some  interdependent.  By 
performing  an  analysis  using  each 
mode,  it  is  possible  to  estimate  a 
value  between  the  two  when 
appropriate.  This  will  result  in  a 
better  definition  of  the  actual 
parameter  variance  for  most  programs. 

Calculations  are  performed  using 
a  Monte  Carlo  simulation.  The  user 
defines  the  number  of  passes  through 
the  algorithm.  Table  1  shows  the 
expected  average  error  in  generating 
random  numbers  as  a  function  of  number 
of  passes  for  the  IBM  system.  The 
random  numbers  are  seeded  by  using  the 
minute  and  second  values  of  the 
internal  clock  function.  The  random 
function  for  the  Apple  system  does  not 
require  explicit  seedinq.  Tests  show 
the  same  level  of  expected  error  for 
the  Apple  as  for  the  IBM. 
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RISK  CALCULATION  FILE  NAME  IS  ?  MASSFIL 

RISK  PARAMETER  IS?  MASS 

IS  THIS  THE  DESIRED  RISK  PARAMETER  ?  ? 

SELECT  SIMULATION  MODE 

1 - DEPENDENT  RISKS 

2-  INDEPENDENT  RISKS 
USE  MODE  ?  2 

NUMBER  OF  PASSES’  200 


Example  5  Prompts  for  Calculation  Mode 


Table  1  Average 

Error  in  Random 

Number  Generation 

#  Passes 

*  error 

50 

6.2 

100 

5.4 

200 

3.5 

500 

1.2 

1000 

0.6 

The  required  execution  time  is  a 
■function  of  the  number  of  risk 
records,  the  type  of  probability 
density  function,  and  the  specified 
number  of  passes.  Test  cases  run 
indicate  execution  time  of  about  1 
second  per  pass  on  the  IBM  FC  and 
slightly  longer  on  the  APPLE  II  system 
when  using  interpreter  BASIC.  Using  a 
BASIC  compiler  with  the  IBM  decreased 
run  time  by  4  1/2  times  for  an 
identical  case.  In  addition,  the 
advances  being  made  in  speed  of 
processor  chips  for  microcomputers 
will  further  decrease  execution  time 
in  the  future. 

After  calculations  are  completed 
the  range  of  the  parameter  and 
cumulative  probability  function  are 
output  in  tabular  and  plotted  form. 
After  each  calculation  and  output  is 
completed,  the  user  is  permitted  to 
calculate  compound  risk  effects  by  the 
prompts  shown  in  Example  6. 
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Example  6  Risk  Convolution  Prompts 
II. 3  RAMP  Applications 

Fallowing  are  some  examples  of  how 
RAMP  has  been  used  to  determine  a 
range  of  values  for  technical 
parameters,  manpower  requirements,  and 
program  costs. 

Evaluating  Risk  of  Staying  Within  Mass 
Budget 

Figure  3  shows  4  component  mass 
estimates  and  the  resulting  cumulative 
probability  function  obtained  by  RAMP 
execution.  The  total  mass  budget  wa^s 
190  kilograms  resulting  in  an  initial 
risk  of  257..  As  the  program  is 
performed  the  mass  estimates  are 
continually  updated  and  the  mass  risk 
recomputed.  The  risk  can  be  plotted 
vs.  time  as  shown  in  Figure  4. 

Ideally,  the  risk  should  decrease  as 
the  program  matures.  If  not, 
management  action  can  be  pursued 
earlier  than  would  be  probable  without 
benefit  of  the  quantitative  risk 
assessment.  Similar  analyses  can  be 
performed  for  any  technical  parameter 
to  predict  and  monitor  technical  risk. 
As  will  be  discussed  later,  the 
results  can  be  normalized  to  evaluate 
overall  program  risk. 

Quantitative  risk  assessment  can 
also  improve  the  ability  to  tradeoff 
alternative  designs.  Figure  5  shows 
the  risk  data  and  predicted  mass  for  a 
case  where  the  mass  estimates  for 
components  3  and  4  are  changed  to 
represent  a  different  system  design. 
For  the  new  design  the  initial  mass 
risk  is  <5  7..  Now,  however,  other 
factors  such  as  increased  complexity 
must  be  traded  off  as  well. 


Evaluating  State-of-the-Art (SOA)  Risk 

Figure  6  shows  the  result 
obtained  for  predicting  a  range  of 
technical  complexity  figures  for  input 
to  a  cost  estimating  model.  (This 
points  out  a  valuable  use  of  RAMP — its 
use  to  provide  dataref lecti ng  program 
uncertainties  for  use  in  existing 
tools. > 
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Each  o-f  6  WBS  task  were  analyzed 
to  estimate  relative  complexity 
•factors  on  a  scale  from  0.2  (easy)  to 
3.3  (most  difficult).  Cardinal 
weights  were  also  defined  for  each 
task  to  relate  their  relative 
importance  to  the  program.  These 
values  are  shown  in  Table  2. 


Figure  3  Cumulative  System  Mass  Function 
for  Initial  Design  Configuration 
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Figure  4  Risk  of  Meeting  Mass  Budget 
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Figure  5  .Cumulative  System  Mass  Function 
for  Alternative  Design 


Table  2  Technical  Complexity  Risk  Data 


WBS 

Weight 

Low 

Most  Likely 

High 

1.0 

4 

0.2 

0.3 

0.4 

2.0 

8 

0.7 

0.8 

0.9 

3.0 

12 

0.2 

0.2 

0.3 

4.0 

3 

0.6 

0.6 

0.8 

5.0 

3 

1.0 

1.0 

1.0 

6.0 

3 

0.4 

0.4 

0.6 

Figure  6  Cumulative  Function  for  Assessing 
Program  Technical  Complexity 


Estimating  Engineering  Labor  Cost 

A  method  was  desired  to  estimate 
engineering  labor  cost  considering 
uncertainty  in  the  total  hours 
required  to  perform  a  set  of  tasks  by 
different  labor  grade  mixes.  Figure  7 
shows  the  probable  direct  cost 
obtained  from  the  estimates  of  Table 
3.  The  labor  burden  density  function 
was  applied  to  this  result  to  give  the 
burdened  co^t  estimate  shown  in  Figure 
0.  Finally,  the  GLA  density  function 
was  applied  resulting  in  the  total 
probable  cost  function  shown  in  Figure 
9.  This  type  of  analysis  provides 
data  for  a  rational  selection  of  price 
of  labor. 
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Table  3  Probability  Density  Function  Data 
for  Manpower  Cost  Estimate 
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Figure  9  Cumulative  Cost  Function  With  G&A 
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Figure  7  Unburdened  Manpower  Cost  Function 
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Figure  8  Burdened  Manpower  Cost  Function 


Evaluating  Program  Performance  Risk 

Tradeoff  of  alternative  program 
approaches  requires  evaluation  of 
overall  risk  based  on  allocated 
resources.  A  typical  risk  analysis 
scenario  first  uses  the  risk 
assessment  methodology  to  predict 
cumulative  schedule  and  cost 
requirements.  Consider  the  case 
represented  in  Figure  10.  Using  RAMP, 
an  initial  cost  prediction  was 
obtained  for  the  listed  6  top  level 
functions.-  A  baseline  price  of  $86M 
was  selected  based  on  results.  The 
S86M  was  then  allocated  to  the  tasks 
as  shown  under  column  "V".  These 
allocated  values  were  then  used  to  go 
back  into  the  data  base  to  determine  a 
risk  for  each  task  based  on  its 
allocation.  The  resultant  risk  for 
each  task  and  an  overall  weighted  risk 
for  the  program  were  calculated  and 
output  as  shown.  It  now  is  easy  to 
respread  the  initial  allocation  to 
reduce  risk  for  individual  tasks  or  to 
minimize  overall  risk. 

Another  way  RAMP  can  be  used  to 
predict  overall  program  risk  is  to  use 
estimates  of  probability  of  success 
from  program  task  leads  after  initial 
resource  allocation.  Each  lead 
provides  low,  modal,  and  high 
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Evaluating  Schedule  Risk 
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Figure  10  Evaluation  of  Overall  Program 
Risk  After  Allocating  Funds 


While  RAMP  does  not  include 
ability  to  create  network  models,  it 
can  be  used  to  perform  first  order 
schedule  risk  analyses  or  to  simplify 
network  models.  Figure  11  shows  a 
simple  program  schedule  network.  RAMP 
could  be  used  to  determine  an 
equivalent  cumulative  function  for 
each  loop  resulting  in  the  simplified 
network  shown  in  Figure  12.  The  final 
step  would  be  to  use  RAMP  to  determine 
a  cumulative  function  for  both 
remaining  paths  and  using  the  results 
to  predict  risk  of  meeting  the  program 
milestone.  For  programs  in  early 
stages  this  might  provide  a  means  to 
reduce  effort  required  when  program 
hierarchies  change  frequently,  as  is 
common  with  high  technology  programs. 


probability  values.  A  weighting 
factor  reflecting  task  importance  is 
assigned  by  the  manager  with  the  sum 
of  weights  equal  to  1 .  In  RAMP,  these 
estimates  are  entered  as  performance 
risks  with  a  required  parameter  name 
"NORMED".  The  calculation  produces  a 
cumulative  program  risk  combining 
different  types  of  risk  parameters. 


Figure  11  Initial  Network  Model 


Figure  12  Reduced  Network  Model 
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IV.  Summary 


RAMP  was  developed  to  meet  a 
•functional  requirement  for  a  risk 
assessment  methodology  that  can 
readily  used  to  improve  planning  and 
management  of  acquisition  programs. 

The  methodology  emphasizes  acquiring, 
maintaining,  and  communicating  risk 
data  to  aid  both  the  customer  and 
contractor  managers.  The  calculation 
capabilities  of' RAMP  could  be 
supplemented  by  more  esoteric 
algorithms  if  so  desired.  The  present 
capabilities  are  considered  adequate 
for  now. 

The  risk  assessment  methodology, 
including  RAMP,  is  currently  being 
used  on  several  programs  and  is  being 
included  in  several  proposals  as  the 
approach  to  be  followed  to  perform 
risk  assessment.  The  examples 
presented  here  represent  only  a 


fraction  of  the  potential  application 
to  program  management.  The 
methodology  can  be  applied  to  any 
program  that  needs  to  determine 
effects  of  program  uncertainties.  The 
ease  of  use  of  RAMP  coupled  with  the 
low  cost  and  portability  of 
microprocessors  make  it  an  attractive 
alternative  to  previously  developed 
risk  assessment  programs.  It  may  also 
serve  to  increase  acceptance  of  the 
use  of  quantitative  assessment 
techniques  by  managers  by  making  them 
more  transparent  to  the  user. 
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INTRODUCTION 


J  The  function  of  the  Directorate  of  Cost 
Analysis  at  the  Aeronautical  Systems  Division  . 
(ASD)  of  the  Air  Force  Systens  Command "(AF5C)~V 
Is  to  prepare  and  present  cost  estimates  for  ' 
weapon  systems  in  various  stages  of  acquisi¬ 
tion.  Frequently  these  estimates  take  the 
form  of  Independent  Cost  Analyses  -4TCAs-)— P 
performed  to  test  the  reasonableness  of  the 
program  office  estimate  or  Most  Probable  Costs 
-(MFCs )-  developed  in  support  of  Source 
Selection  Evaluations. 

Perhaps  the  most  significant  problem  In 
predicting  such  future  costs  is  the  risk  and 
uncertainty  Inherent  In  predicting  the  future. 
Although  ASD  has  not  had  a  breakthrough  In 
measuring  the  cost  Impact  of  risk  and  uncer¬ 
tainty,  a  number  of  approaches  have  been 
developed,  adopted,  or  refined  to  address  risk 
In  cost  estimating.  These  techniques  or 
methods  which  have  been  incorporated  into  cost 
estimating  are  as  follows: 

Learning  Curve  Adjustments, 

Technology  Indexing,  „  ,  ■. 

0rJ*O 

EC0AModel, 

Proposal  Analysis, 

Range  of  Estimates, 

Confidence  IndexeSj  a-nJ. 

Risk/Uncertainty  Assessments  , 

These  several  techniques  will  be  addressed 
below. 


will  flatten  the  slope.  This  is  a  commonly 
accepted  effect  and  has  been  addressed  in  the 
pertinent  literature.  In  addition,  develop¬ 
ment  hardware  is  manufactured  in  a  prototype- 
development  environment  with  associated 
disruptions  because  of  constant  change  based 
upon  feedback  from  the  design,  development  and 
test  effort.  Both  the  low  rate  and  prototype 
environment  create  a  situation  with  less 
opportunity  to  learn  and  greater  opportunity 
to  forget. 

An  additional  effect  which  serves  to 
flatten  learning  curve  slopes  is  concurrency. 
Concurrency  is  defined  as  an  overlap  between 
production  and  development.  To  the  extent 
that  production  occurs  prior  to  flight  test 
completion  there  will  be  increased  Engineering 
Change  Orders  (ECOs)  to  accommodate  configu¬ 
ration  revisions.  A  high  change  volume  will 
likely  be  generated  by  the  simultaneous  "build 
and  test"  environment.  Concurrency  also  has 
a  retarding  effect  upon  production  learning. 
Not  only  will  learning  disruption  occur  on 
those  Items  or  elements  being  directly 
affected,  but  there  Is  a  collateral  Impact  on 
those  areas  which  are  not  directly  Involved 
In  the  changes. 

For  several  years  ASD  has  been  using 
adjustments  to  the  learning  curve  to  reflect 
and  account  for  these  aspects  of  program  risk 
and  uncertainty.  Specifically  learning  curves 
are  flattened  for  concurrent  programs  where 
disruptions  are  high,  production  rates  low, 
and  rework  potential  significant.  A  definite 
quantifiable  relationship  has  been  shown  to 
exist  between  prototype  and  production  Tj's 
(first  '  unit  costs).  In  particular  this 
relationship  has  been  studied  and  documented 
extensively  by  Tecolote. 


LEARNING  CURVE  ADJUSTMENTS 

The  first  place  an  analyst  nust  address 
risk /uncertainty  is  within  the  basic  estimate. 
For  several  years  ASD  has  been  using  adjust¬ 
ments  to  the  learning  curve  to  reflect  and 
account  for  certain  aspects  of  risk  and  uncer¬ 
tainty. 

Development  In  general  is  found  to  con¬ 
tain  flatter  learning  curves  than  follow-on 

production.  There, are  several  reasons  for 
this.  First,  development  programs  are  for 
the  most  part  small  quantities  of  hardware 
produced  over  a  several  month  or  perhaps 
several  year  p?r1od.  This  low  production  rate 


Figure  1  shows  an  example  of  how  the 
manufacturing  labor  was  estimated  for  the 
KC- 135  Re-Engine  Modification  ICA.  The  com¬ 
posite  Tj  value  Is  in  hours  and  production 
learning  curves  emanate  f-om  It.  The  Tj  was 
based  on  current  contractor  experience  on 
analogous  programs  which  were  very  similar 
in  weight,  scope  and  complexity  of  effort. 
The  underlying  learning  curve  was  calculated 
using  the  same  actuals  used  to  calculate  Tj. 
However,  to  take  Into  consideration  the  slower 
rate  of  learning  which  would  be  occurring 
during  the  period  of  concurrency,  the  initial 
production  quantities  were  projected  on  the 
flatter  learning  curve  slope  shown.  It  was 
felt  that  three  months  after  flight  test 
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completion  all  changes  would  be  final  with 
concurrency  no  longer  impacting  production. 
It  was  at  this  point  in  time  that  twenty  five 
units  would  be  complete  or  in  .process  and 
would  be  affected  by  a  lessening  of  learning 
due  to  the  change-traffic  and  manufacturing 
concurrent  with  development  and  flight  test. 
Therefore  this  flatter  slope  was  extended 
through  the  twenty-fifth  unit.  The  degree  of 
flattening  from  the  underlying  slope  (6  per¬ 
centage  points)  was  based  upon  judgment  as 
well  as  ASO  experience.  Beginning  with  the 
twenty-sixth  unit  the  estimate  would  pick  up 
where  the  underlying  learning  curve  woula  have 
been  if  there  has  been  no  concurrency  between 
development  and  production.  From  that  point 
on,  the  manufacturing  labor  estimate  proceeds 
down  the  underlying  learning  curve. 

Figure  2  portrays  the  manufacturing  labor 
methodology  of  the  recent  B-1B  ICA.  The  first 
four  B-1A  aircraft  were  test  aircraft  manu¬ 
factured  in  a  prototype  environment.  The 
actual  costs  of  these  four  aircraft  were  used 
to  derive'  the  prototype  T^.  Because  of  the 
production  gap  between  these  aircraft  and  the 
B-1.8  production  a  complete  loss  of  learning 
was  anticipated.  Because  the  initial  phase  of 
thu  B-1B  production  will  also  be  in  a  proto¬ 
type  environment,  this  T^  was  used  as  the 
first  production  aircraft  estimate.  The  early 
phase  of  B-1B  production  will  occur  simul¬ 
taneous  with  the  establishment  of  a  viable 
production  line  Including  extensive  tooling 
build-up/lmplementation,  facilities  moderni¬ 
zation,  and  capital-type  rehabilitation.  This 
will  be  a  period  of  turmoil  for  the  first  few 
B-lBs  produced.  In  addition  the  B-lBs  will  be 
produced  at  a  relatively  low  production  rate 
with  relatively  long  flow  times.  It  was  esti¬ 
mated  that  this  prototype  environment  will 
affect  the  first  nine  aircraft,  therefore  this 
prototype  Tj  was  extended  on  the  learning 
curve  shown  through  unit  nine. 

Beginning  with  unit  ten  the  estimate 
drops  to  an  underlying  slope  (same  learning 
curve  slope  but  lower  Tj)  which  projects  from 
a  production  Tj.  This  production  Ti  was 
derived  from  a  separate  methodology  than  the 
prototype  T^  and  assumes  a  full  production 
capability  and  environment.  The  prototype- 
production  relationship  (production  Tj  is 
approximately  80%  of  prototype  Ti)  is  con¬ 
sistent  with  the  Tecolote  research  previously 
referenced  as  well  as  ASO  experience. 

If  the  B-1B  were  to  proceed  into  a  full 
production  environment  the  manufacturing  labor 
would  be  projected  from  the  production  Tj  down 
the  underlying  slope  which  is  the  steeper  of 
the  two  shown  in  Figure  2,  However,  because 
the  B-1B  program  Is  50%  concurrent  the  under¬ 
lying  slope  will  not  be  attained  until  unit 
twenty  six.  In  other  words  between  unit  nine 


and  twenty  five,  the  labor  estimate  is  pro¬ 
jecting  from  a  production  Tj  on  a  concurrency- 
flattened  slope.  As  apparent  from  the  figure 
the  first  nine  "prototype-environment" 
aircraft  are  also  on  this  concurrency-driven 
flatter  slope.  Normally  with  a  100%  con¬ 
current  program,  the  concurrent  slope  would  be 
6%  flatter  than  the  underlying  slope.  But  In 
this  case  the  flattening  is  only  3%  because 
concurrency  is  50%  rather  than  100%.  Percent 
concurrency  is  measured  as  the  number  of 
months  between  production  contract  award  and 
flight  test  completion  divided  by  the  number 
of  months  between  development  contract  award 
and  flight  test  completion.  In  the  case  of 
B-l,  concurrency  would  have  been  100%  except 
for  the  fact  that  previous  development  had 
occurred  and  a  large  percent  of  the  aircraft 
was  common  design.  Thus  the  denominator  of 
the  concurrency  equation  was  increased  to 
reflect  the  previous  development  months  of 
the  common  design. 

TECHN0L0GV  INDEXING 

TEMPER  COST  is  an  acronym  which  refers  to 
a  temporal,  performance  and  cost  methodology 
which  allows  an  analyst  to  relate  cost  to 
increases  in  technology  (performance)  over 
time.  There  are  several  major  assumptions  or 
premises  which  need  to  be  established  before 
explaining  this  method.  First  -  technology 
growth  occurs  over  time.  Second  -  performance 
is  one  of  several  proxies  for  technology. 
Third  -  performance  (technology)  increases 
over  time  at  some  growth  rate.  Fourth  -  an 
Increase  in  performance  over  time  equates  to 
one  of  three  conditions  as  relates  to  cost: 
1  -  at  the  "given11'  technology  growth  rate 
over  time  greater  performance  is  achievable 
at  equal  cost  (to  todays  performance),  2  -  at 
less  than  the  "given"  growth  rate,  greater 
performance  Is  achievable  at  lower  cost,  3  - 
at  greater  than  the  “given"  growth  rate, 
greater  performance  is  achievable  at  higher 
cost. 

Figure  3  graphically  portrays  this  con¬ 
cept.  A  technology  index  is  shown  along  the 
Y  or  dependent  axis  with  time  along  the  X  or 
Independent  axis.  The  premise  of  course  is 
that  technology  growth  or  our  proxy,  perfor¬ 
mance,  is  a  function  of  time.  This  tech¬ 
nology  growth  as  a  function  of  time  is  a 
result  of  total  national  resources  (govern¬ 
ment,  industry,  scientific,  etc.)  being 
brought  to  bear  over  time.  It  is  assumed  that 
these  resources  are  so  large  in  total  that  no 
one  single  effort  (such  as  a  large  government 
RSD  effort)  can  change  the  position  of  the 
technology  growth  line.  As  technology  growth 
compounds  over  time  along  this  line,  the  tech¬ 
nology  reached  over  the  time  continuum  is 
available  at  equal  cost  (excluding  inflation). 


If  l'n  four  years  a  technology  growth  or 
performance  Increase  of  2.0  is  required  and 
the  technology  growth  will  be  at  2.5  plus  as 
shown  in  Figure  3,  then  the  required  tech¬ 

nology  of  2.0  will  be  available  at  a  lower 
cost  than  current  technology.  As  a  matter  of 

fact  the  technology  required  in  four  years 

will  be  available  in  slightly  less  than 

three  years  (along  the  equal  cost-technology 
growth  line).  Therefore  the  difference 
between  the  required  technology  of  2.0  and  the 
technology  growth  line  at  4  years  represents 
a  cost  benefit  or  savings  over  current  levels 
of  technology. 

On  the  other  hand  if  3.5  is  the  required 
technology  level  in  that  same  time  period,  but 
the  technology  has  not  yet  arrived  (along  the 
technology  growth  line),  then  it  is  achievable 
only  at  a  cost  penalty  or  premium  as  shown. 
Only  in  5  1/2  years  could  the  3.5  level  of 
technology  be  available  at  equal  cost,  i.e. 
that  technology  will  arrive  in  5  1/2  years. 

The  greatest  uncertainty  in  this  method 
is  probably  the  determination  of  the  rate  of 
technology  growth,  as  shown  bounded  by  the 
dashed  lines  emanating  from  the  origin  on  the 
graph. 

A  hypothetical  example  using  Figure  4 
will  further  clarify  this  concept  and  show  how 
the  analysis  was  used  In  a  recent  estimate. 
System  Z  is  a  current  program  using  1983 
technology  to  achieve  a  performance  normalized 
to  a  value  of  1.0.  Actual  production  costs 
are  available  for  this  program.  It  is  now 
required  that  the  production  costs  for  a  very 
Similar  system,  system  Z-4,  be  estimated. 
System  Z-4  requires  a  performance  of  3.0 
(normalized  to  Z  performance).  This  increased 
performance  (read:  technology)  is  required  in 
1987,  four  years  hence.  The  Industry  has  been 
Surveyed  and  the  growth  rate  for  this  tech¬ 
nology  or  performance  is  estimated  to  be  201 
per  year  based  on  past  trends  and  growth 
patterns.  Plotting  this  given  technology 
growth  on  Figure  4  results  in  the  205  ISO-Cost 
Curve  shown.  Given  this  205  growth,  a  3.0 
performance  Increase  will  arrive,  as  shown.  In 
6  years.  This  calculates  as  the  log  of  3.0 
divided  by  the  log  of  1,2.  Of  course,  given 
the  theoretical  construct,  to  acquire  such  a 
capability  sooner  will  cost  more  than  current 
performance  costs  and  to  acquire  It  later  will 
cost  less.  At  the  time  we  need  the  perfor¬ 
mance  Increase,  i.e.  4  years  hence,  at  the  205 
growth  rate,  equal  cost  technology  will  only 
have  achieved  a  performance  Increase  of  2.0. 
The  performance  Increase  required,  3.0, 
divided  by  the  performance  Increase  achievable 
at  the  required  time  (at  equal  cost),  2.0, 
yields  a  1.5  TEMPER  Cost  factor.  Therefore  to 
achieve  this  capability  two  years  “ahead  of 
its  time"  will  cost  505  more  than  the  current 
technology.  .The  application  of  this  1.5  fac¬ 
tor  to  the  cost  of  System  Z  will  result  in  our 


best  estimate  of  System  Z-4  in  four  years, 
excluding  the  impact  of  inflation  and  assuming 
Z-4  and  Z  are  identical  except  for  the  perfor¬ 
mance  increase. 

ECO  MODEL 

ASD  has  developed  an  ECO  (Engineering 
Change  Order)  Model.  This  model,  previously 
called  the  ASO  Unencumbered  Funds  Model, 
estimates  ECOs  and  management  reserve  (MR) 
required  for  a  program.  These  ECOs  and 
management  reserve  are  funds  for  unantici¬ 
pated,  but  within  scope  requirements.  This 
model  provides  a  systematic  approach  which 
enhances  estimator  judgment  in  arriving  at  an 
ECO /MR  estimate.  Although  judgment  is  still 
required  to  establish  the  input  parameters, 
that  judgment  is  now  in  the  context  of  a 
consistent,  logical  and  procedural  process. 

Ideally  a  statistically  derived  cost 
estimating  relationship  (CER)  would  be  derived 
which  quantitatively  related  EC0/MR  cost  to 
pertinent  independent  variables.  However, 
the  unique  nature  of  EC0/MR  funds  preclude 
this  type  of  cost  method.  Subsequent  to  a 
program's  completion  no  historical  data  is 
available  which  identifies  the  required 
EC0/MR.  Funds  set  aside  for  such  contin¬ 
gencies  are  later  allocated  to  whatever  use 
they  are  needed  and  thus  appear  within  the 
respective  cost  element  in  the  actual  record 
of  cost.  On  the  other  hand  Initial  estimates 
may  have  Included  estimates  for  such  contin¬ 
gencies,  either  explicitly  or  Implicitly.  To 
further  compound  this  problem  very  few  data 
points  exist  while  a  large  number  of  causative 
factors  can  De  addressed.  Thus  a  CER  approach 
to  estimate  EC0/MR  could  not  be  used. 

The  data  base  used  for  the  generation  of 


this 

system 

model  includes  the 
programs: 

following  weapon 

F- 15 

C-5A 

F-5E 

F-15  Nav  Subsys 

F-16 

F-lll 

SRAM 

B-52  SRAM  MOD 

A-10 

F-4E 

MAVERICK 

F-15  FCS 

B-52  Phase  VI  ECM  Mod 

F-15  Avionics 

Again 

the  data 

provided  by 

these  systems  was 

sketchy  and  subject  to  the  limitations  stated 
above.  Based  on  this  data  as  well  as  dis¬ 
cussions  with  cost  analysts,  engineers,  and 
program  managers  the  factors  affecting 
unencumbered  funds  were  identified.  The  iden¬ 
tified  factors  whicn  drive  the  magnitude  and 
direction  of  EC0/MR  are  shown  below. 

1.  Degree  of  Certainty  of  Configuration 
at  Time  of  Contract  Award 

2.  Concurrent  Development  and  Production 
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3.  Risk  Due  to  Technical  Advancement 

4.  Program  Length 

5.  Multi-Service  Participation 

6.  Multi-Purpose  Systems 

7.  Contractor  Proficiency 

8.  Scope  of  Total  Production  Program' 

9.  Development  Schedule  Compression 

From  this  set  of  applicable  factors  it 
was  determined  that  two  separate  and  distinct 
common  denominators  existed,  program  com¬ 
plexity  and  program  schedule  impact.  An 

attempt  was  made  to  derive  a  composite  measure 
of  complexity  from  the  appropriate  factors. 
In  addition  the  concurrency  between  develop¬ 
ment  and  production  was  quantified  as  well 
as  the  impact  of  a  compressed  development 
schedule. 

A  group  of  ASD  analysts  reviewed  all  the 
data  from  the  systems  previously  listed  and 
discussed  the  aforementioned  factors  and  how 
these  logically  relate  to  ECO/MR.  They  then 
made  several  composite  judgment  estimates  for 
ECO/MR  as  a  percent  of  production  costs,  as 
a  function  of  program  complexity,  schedule 
impact  and  overall  program  nature.  The 
established  cause  and  effect  logic  was  ther 
extended  to  develop  step-by-step  procedures 
for  estimating  ECO/MR. 

The  other  consideration  In  the  develop¬ 
ment  of  the  ECO  Model  was  that  the  nature  of 
the  data  led  the  analyst  to  a  procedure  which 
separately  addresses  New  Aircraft  Systems,  New 
Avionics  Systems  and  Aircraft  Modification 
Programs. 

The  first  step  in  establishing  an  ECO 
estimate  using  the  ECO  Model  Is  to  complete 
the  Complexity  Score  Computation  Worksheet 
(Table  1)  This  worksheet  develops  a  program 
complexity  score  (S)  by  summing  scores  for  six 
individual  categories  of  "pertinent"  factors. 
Each  factor  Is  scored  by  matching  the  program 
characteristics  to  the  most  appropriate 
description  for  each  factor.  These  worksheets 
should  be  completed  cat^d  on  discussions  and 
inputs  from  cost  analysts,  engineers  and 
program  managers. 

The  second  step  Is  to  establish  the  per¬ 
cent  concurrency  between  development  and  pro¬ 
duction.  This  percent  concurrency  is  the 
number  of  months  between  first  production 
contract  award  and  completion  of  flight  tests 
divided  by  the  number  of  months  from  develop¬ 
ment  (FSD)  contract  award  to  completipn  of 
flight  test.  This  calculated  percent  shows 
the  percent  of  development  whic£  Is  overlapped 
by  production. 


The  third  step  is  to  determine,  based  on 
judgments  from  engineers  and  program  managers, 
if  the  development  schedule  is  compressed. 

Step  four  entails  going  to  Table  2  with 
the  results  of  each  of  the  first  three  steps. 
Table  2  shows  the  "UF  Requirement  Percent 
Program  Data"  for  New  Aircraft  Systems.  As 
mentioned  earlier  there  is  a  different  table 
for  each  of  thre6  different  types  of  systems/ 
programs.  Using  the  complexity  score,  S,  as 
well  as  the  percent  concurrency  either  under 
the  normal  or  compressed  schedule  column,  an 
ECO/MR  percent  can  be  read  from  the  table. 
(As  shown  in  Table  2  the  percent  concurrency 
from  either  column  can  be  translated  into  a 
"t"  value  which  when  combined  with  the  "S" 
value  will  give  an  ECO/MR  percent  from  the 
table. ) 

The  final  step  is  to  estimate  the  total 
unencumbered  funds  dollar  requirement  by 
multiplying  the  percent  ECO  factor  by  the 
recurring  production  costs.  Finally  the 
ECO/MR  funds  which  have  been  estimated  need  to 
be  spread  by  fiscal  year.  This  fiscal  year 
spreading  requires  estimator  judgment  as  well 
as  awareness  of  program  peculiar  character¬ 
istics.  In  general  the  ECO  funds  will  show  a 
decrease  as  a  percent  of  recurring  production 
each  year.  Usually  the  first  year  percent  is 
lower  than  the  second  year.  Yearly  rates  must 
be  established  by  trial  and  error  in  order  to 
maintain  the  integrity  of  the  total  percent 
calculated  as  well  as  following  a  logical 
decreasing  profile. 

PROPOSAL  ANALYSIS 

During  a  Source  Selection  Evaluation  an 
independent  Cost-To-The-Government  panel  makes 
an  independent  estimate  of  the  contractor 
proposal.  This  estimate  is  called  an  MPC  or 
Most  Probable  Cost.  This  MPC  for  the  instant 
contract  must  be  analyzed  in  light  of  the 
proposed  values  and  with  consideration  of  the 
ECO  Model  just  covered.  Table  3  shows  several 
possible  scenarios  for  an  FPI  contract.  If 
the  MPC  exceeds  the  ceiling  of  the  proposal, 
which  of  course  is  the  limit  of  government 
liability  for  that  potential  contract,  the  MPC 
would  have  to  be  adjusted  down  to  that  ceiling 
value.  However,  assuming  the  MPC  is  accurate 
and  represents  the  cost  which  will  actually  be 
incurred,  the  contractor  would  be  buying-in 
and  using  his  own  funding  to  pay  for  the 
costs  incurred  ir  excess  of  ceiling.  The  risk 
here  is  that  the  contractor  will  attempt  to 
get  well  or  recoup  his  losses  in  follow-on 
efforts.  So  there  is  a  risk  that  he  will 
"roll-over"  these  costs  to  later  phases  or 
contracts.  To  the  extent  that  it  can  be 
quantified  these  "additional"  costs  should 
be  reflected  on  estimates  of  the  follow-on 
activities.  Even  with  an  FFP  contract  where 
ceiling  is  target,  a  contractor  can  buy-in 


with  hopes  of  getting  well  either  on  follow-on 
effort  or  if  out-of-scope  changes  are  nego¬ 
tiated  to  the  FFP  contract. 

If  the  MPC  falls  below  the  proposed 
target  value  then  it  would  be  wise  to  re¬ 
evaluate  the  MPC  to  determine  why  it  falls 
short  of  the  target.  It  would  also  be  appro¬ 
priate  to  review  the  contractor's  past  perfor¬ 
mance  on  relevant  contracts.  For  example,  if 
the  contractor  has  consistently  come  in  under 
target  and  if  the  MPC  is  within  the  range  of 
these  final  contract  values  (as  a  percent  of 
target),  then  the  MPC  could  be  used  without 
adjustment.  The  converse  is  of  course  also 
true,  i.e.  an  adjustment  of  MPC  to  target  is 
appropriate  if  the  contractor  rarely  if  ever 
falls  under  target. 

Most  likely  the  MPC  will  fall  somewhere 
between  target  and  ceiling.  In  this  case  the 
MPC  is  indicating  that  the  final  cost  will  be 
at  this  value  above  target  but  below  ceiling. 
Again  past  performance  should  be  analyzed  as 
a  test  of  the  reasonableness  of  the  MPC  in 
relation  to  target/celling  values.  It  Is  also 
here  that  the  ECO  Model  can  play  a  role  in 
the  proposal  evaluation.  Specifically  the 
difference  between  the  MPC  and  the  contract 
target  could  be  used  to  estimate  the  addi¬ 
tional  ECO/MR  required  as  a  result  of  the 
unrealistically  low  proposal  value,  i.e.  the 
buy-in.  Where  a  proposal  is  being  evaluated 
without  benefit  of  an  MPC  estimate  additional 
ECO  costs  can  be  developed  by  using  the  “Other 
Factors"  category  of  the  ECO  Model.  An 
approach  would  be  to  run  the  ECO  Model  with 
and  without  the  “buy-in"  consideration,  the 
delta  being  the  estimated  impact  of  the 
"buy-in."  This  represents  one  facet  of  ECO 
while  the  balance  of  the  ECO  is  for  "normal 
requi rements.  “ 

In  all  cases,  contract  share  ratios 
should  be  considered  in  deriving  final  MPC 
values,  i.e.  the  MPC  must  be  consistent  with 
the  contractual  terms  and  conditions  of  the 
proposal . 

RANGE  OF  ESTIMATES 

Most  frequently  in  cost  estimating,  point 
estimates  are  required,  whether  for  budgeting 
purposes  or  to  test  the  reasonableness  of 
program  office  estimates.  These  point  esti¬ 
mates,  as  most  probable  costs  {thus  the  "MPC" 
of  source  selection),  have  a  very  small  chance 
of  being  the  right  number;  rather  they  repre¬ 
sent  the  50%  probability  position  in  a  range 
or  band  of  estimates.  For  example  if  the 
MPC  or  ICA  estimate  is  7.4  billion  dollars, 
what  is  really  meant  is  that  there  is  a  50% 
probability  of  the  cost  being  less  than  7.4 
and  a  50%  probability  of  the  cost  being 
greater  than  7.4.  However,  without  some 
bounding  ofr  the  range  or  band  we  have  little 
indication  of  the  uncertainty  surrounding  7.4 


billion,  i.e.,  is  there  a  90%  probability  that 
the  cost  will  be  between  7.0  and  7.8  billion 
or  does  the  90%  probability  fall  between  6.0 
and  9.0  billion.  The  latter  estimate  of  7.4 
has  more  uncertainty  than  the  former  estimate 
of  7.4.  A  great  deal  has  been  written  about 
confidence  intervals  around  estimates  but  very 
little  has  been  put  into  practical  use. 

An  estimating  range  around  a  point  esti¬ 
mate  can  provide  a  quantification  of  the 
uncertainty  which  surrounds  that  point  esti¬ 
mate.  This  estimating  range  need  not  be 
related  to  a  statistical  probability  of 
specific  value.  On  sever?!  recent  Independent 
Cost  Analyses  { ICAs )  an  estimating  band  of 
reasonableness  was  established.  To  the  extent 
the  program  office  estimate  (being  tested 
for  reasonableness)  fell  within  the  band, 
the  program  office  estimate  was  considered 
reasonable.  Whereas  the  point  estimate 
involves  a  set  of  fixed  assumptions,  a  range 
of  estimates  or  band  of  reasonableness  can 
allow  certain  critical  assumptions  to  vary  and 
thus  depict  the  cost  sensitivity  of  differing 
values  for  these  critical  assumptions.  For 
example  the  C-5B  ICA  had  a  Range  of  Expected 
Costs  based  upon  several  different  assumptions 
-  an  FFP  constrained  estimate,  an  estimate 
with  an  additional  2%  ECO,  and  estimates  with 
two  levels  of  abnormal  inflation.  Table  4 
portrays  these  estimates.  Although  an  ICA 
point  estimate  of  8513  was  established,  the 
range  of  varying  assumptions  resulted  in  a 
band  of  reasonableness  from  7986  to  8695  with 
the  program  office  estimates  {FY84  President's 
Budget)  falling  within  the  reasonableness 
band.  Traditionally  the  ICA  value  of  8513 
compared  to  program  office  estimate  of  8284,  a 
difference  of  less  than  3%,  would  result  in 
the  conclusion  that  the  8284  is  a  reasonable 
estimate. 

The  HR  Maverick  ICA  developed  a  band  of 
reasonableness  based  on  varying  one  critical 
assumption,  .  competition  scenario.  The  ICA 
estimate  methodology  was  sensitive  to  this 
competition  strategy  in  terms  of  percent 
splits  of  buys  between  the  competitors  and 
buy-out  years  where  the  winner  competes  for 
the  balance  of  FY  buys.  Because  the  program 
office  estimate  was  not  sensitive  to  these 
competition  parameters  (they  assumed  there 
would  be  a  10%  reduction  in  cost  due  to 
competition,  without  regard  to  different 
split  percents)  it  was  impossible  to  choose  a 
competition  scenario  for  the  ICA  to  serve  as 
a  test  of  reasonableness  of  the  program  office 
estimate.  The  only  logical  alternative  was 
to  determine  if  the  program  office  estimate 
was  within  the  band  of  estimates  generated  by 
various  competition  scenarios.  Although  Table 
5  only  shows  three  scenarios  or  strategies 
numerous  were  estimated,  with  the  three  shown 
being  representative  of  low,  middle,  and  high 
values  for  the  range.  Because  the  program 
office  estimate  fell  within  the  range  it  was 
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considered  to  be  a  reasonable  estimate,  and 
In  fact  the  ICA  adopted  it  as  its  own  ICA 
estimate. 

CONFIDENCE  INDEXES 

An  AFSC  Regulation  (AFSCR  173-2)  estab¬ 
lished  as  AFSC  policy  the  use  of  cost  estimate 
quality  ratings  for  cost  estimates.  This 
technique,  developed  at  ASD,  provides  for 
management  the  basis  of  the  estimate  and  the 
r*  confidence  in  the  estimate.  Once  management 

U  understands  the  index  system,  all  estimates 

being  presented  can  readily  be  compared  in 
terms  of  methods  and  data  used,  i.e.  overall 
quality. 

The  confidence  Index  has  two  parts,  the 
methods  used  and  the  data  availability.  The 
methods  used  fall  into  four  categories  from  I. 
Detailed  to  IV.  Parametric  and  Factors.  The 
data  availability  fall  into  seven  categories, 
from  A-Actual  cost  of  significant  quantities 
.for*  system  being  estimated  to  G-Limited  cost 
data  and'  limited  physical  and  technical 
definition.  Therefore  the  range  of  the  con¬ 
fidence  index  can  be  from  “IA",  where  the 
confidence  level  of  the  estimate  is  highest 
to  a  "IVG"  where  the  confidence  Index  is  the 
if  lowest.  There  are  of  course  numerous  con- 

S>-i  fidence  levels  in  between.  When  estimates 
are  thus  rated  by  the  analyst  it  is  clearly 
established  that  not  all  estimates  are  of  the 
same  quality.  A  "IA"  estimate  has  the  least 
uncertainty  being  based  on  detailed  methods 
and  actual  data,  whereas  a  IVG  estimate  has 
the  greatest  uncertainty  being  based  on  para¬ 
metric  methods  with  limited  available  data. 
It  would  be  expected  that  a  "IA"  estimate 
would  have  a  narrow  estimating  range  around  it 
while  a  "IVG"  would  have  a  wide  range  of  esti¬ 
mating  probability  or  error.  Likewise  a  "IA" 
should  have  a  smaller  ECO/MR  percent  than  a 
“IVG"  estimate,  because  of  the  differences  In 
uncertainty.  It  should  be  mentioned  that 
because  of  method/data  differences  between 
development  and  production  estimates,  they  are 
usually  assigned  different  index  ratings. 

The  four  methods  addressed  in  the  Index 
are  described  below. 

Detailed  -  This  method  of  estimating 
Involves  detailed  grassroots  calculations. 
It  Includes  Industrial  engineering  standards 
which  measure  labor  hours  and  dollars  for 
discrete  tasks  at  a  low  level  of  detail. 
Frequently  this  type  of  estimate  Is  done  on 
a  functional  labor  basis,  i.e.  engineering, 
manufacturing,  quality  control  and  tooling, 
for  lower  levels  of  the  work  breakdown  struc¬ 
ture  (WBS). 

-‘••C 

H  Analogous  -  This  method  Involves  drawing 

comparisons  with  similar  systems  where  costs 
have  already  been  Incurred.  •  Costs  of  the 
analogy  are  adjusted  according  to  weight, 


performance,  or  complexity  differences  in 
order  to  estimate  the  cost  of  the  system  being 
estimated. 

Parametric  -  This  method  involves  an 
estimate  that  uses  statistically  derived  cost 
estimating  relationships  which  are  developed 
from  actual  cost  and  performance  data.  Such 
relationships  are  expressed  as  equations  which 
have  cost/hours  as  the  dependent  variable 
and  physical /performance  parameters  as  the 
independent  variables.  Parametric  estimates 
are  usually  performed  at  higher  levels  of 
detail  than  the  detailed  method. 

Factors  -  Cost  factors  are  used  to  esti¬ 
mate  cost  as  a  percent  of  another  cost  based 
upon  similar  systems  or  a  historical  data 
base. 

Realizing  that  an  estimate  will  seldom  be 
composed  of  just  one  of  these  methods,  or  data 
availability  for  that  matter,  the  following 
indexes  have  been  established: 

METHODS  USED 

I.  Detailed 

II.  Detailed  and  Parametric 

III.  Analogous  and  Factors 

IV.  Parametric  and  Factors 

DATA  AVAILABILITY 

A.  Actual  cost  of  significant  quantities 

for  the  system  being  estimated  arrayed  by 

functional  and  WBS  breakout. 

B.  Actual  cost  for  development  hardware 

for  the  system  being  estimated  arrayed  by 

functional  and  WBS  breakout. 

C.  Actual  cost  by  function  or  WBS  for 
analogous  systems. 

D.  Firm  contractors'  proposals  with 
detailed  backup  or  negotiated  prices. 

E.  Contractor  budgetary  estimates  with 
program  office  adders  (factors,  ECO,  manage¬ 
ment  reserve,  etc.) 

F.  Limited  cost  data  but  good  descrip¬ 
tions  of  physical,  technical,  and  performance 
characteristics. 

G.  Limited  cost  data  and  limited  physi¬ 
cal,  technical,  and  performance  descriptors. 

RISK/UNCERTAINTY  ASSESSMENTS 

Most  of  our  comprehensive  estimates 
Include  a  risk  analysis  section  where  areas 
of  risk  and/or  uncertainty  are  as  a  minimum 
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identified  and  where  possible  cost  impacts  to 
the  basic  estimate  are  shown.  Specific  areas 
addressed  may  include  inflation  rates,  over¬ 
head  variations  due  to  fluctuating  business 
base,  methodology  assumptions  such  as  learning 
curve  sensitivities,  competition  Impacts, 
technical-difficulty  assessments,  concurrency, 
material  shortage  potential,  technology 
availability,  schedule  risk,  plant  capacity, 
etc.  These  considerations  may  or  may  not  be 
addressed  in  other  ways  by  the  estimate  as 
discussed  heretofore.  The  Risk  section  of  an 
ICA  briefing  will  usually  contain  several 
charts  which  address  the  Confidence  Index,  a 
chart  showing  sensitivity  to  various  inflation 
rates  which  are  always  a  critical  assumption, 
and  a  chart  or  two  addressing  any  other  areas 
of  concern.  For  example  the  KC-135  Re-Engine 
Modification  ICA  briefing  summarized  risk  as 
follows: 

Development 
Hours  Sufficient 

Purchased  Equipment  -  Existing  Design 
Critical  Design  Review  Completed 
No  Design  Problems  -  State-of-the-Art 
Production 
Engine  Long  Lead 
Concurrency  -  Minimum  Impact 
Overall  Assessment  -  Low  Risk 

This  was  a  program  completing  development 
and  about  to  go  into  production.  A  different 
example  would  be  the  ASALM  (Advanced  Strategic 
Air  Launched  Missile)  Independent  Cost  Study. 
This  program  was  very  early  in  its  develop¬ 
ment  with  only  a  propulsion  system  validation 
having  been  completed. 

The  Technical  Risk  and  Uncertainty  addressed 
In  the  study  are  shown  below. 

Technical  Risk 


Airframe 
High  Temperature 
RCS  Reduction 

Uncertainty 

Carrier  Ai  rcraft 

Force  Structure/Roles 

Ultimate  Performance/Design  Require¬ 
ments 

IOC  To  Be  Specified 

Nature  of  Program  to  be  Directed 

Only  Subsystem  Demo  Validation  Phase 
Di rected 

Limited  Industrial  Base 

A  more  comprehensive  approach  to  address¬ 
ing  risk/uncertainty  is  the  Independent 
Schedule  Assessment  (ISA),  an  independent 
assessment  of  the  reasonableness  of  the 
program  office  master  integrated  program 
schedule.  The  ISA,  performed  as  a  complement 
with  the  ICA,  evaluates  general  schedule 
risk,  identifies  critical  paths,  and  deter¬ 
mines  specific  pacing  schedule  events.  Once 
specific  areas  of  concern  are  identified  the 
potential  for  cost  impact  can  be  quantified. 

The  Next  Generation  Training  ICA  includec 
a  comprehensive  ISA  which  incorporated  as  the 
basis  for  evaluation  a  novel  two-dimensior.a 
risk  assessment.  This  assessment  considers 
two  levels  of  schedule  Impact,  major  anc 
minor.  Each  of  these  Impacts  has  three  levels 
of  probability  of  occurrence,  low,  medium  anc 
high.  This  matrix  shown  below  results  in  si; 
schedule  risk  assessment  ratings  as  shown. 


System 

Integration 

Guidance 

Alr-to-AIr  Anti -Radiation  Homing 
Active  Radar 
Propulsion 

Limited  Integral  Rocket  Ramjet 
Experience 


102 


BIBLIOGRAPHY 


A  specific  rating  assigned  to  an  area  would 
have  to  be  explained  in  terms  of  the  potential 
impact.  In  addition  this  impact  could  also  be 
quantified,  i.e.  if  the  impact 'was  realized 
specific  additional  dollars  would  be  incurred 
by  the  program.  Probability  theory  could  also 
be  used  to  estimate  the  expected  cost  impact 
of  all  the  identified  risks. 

SUMMARY 

There  are  a  number  of  methods  currently 
being  used  in  addressing  risk  and  uncertainty 
in  cost  estimating.  These  techniques  are  by 
no  means  the  final  word  on  risk/uncertainty, 
as  other  ideas  can  and  surely  will  be 
developed.  Perhaps  these  concepts  will  be 
further  refined  or  even  serve  as  the  seeds  for 
creating  new  approaches  to  deal  with  risk  and 
uncertainty  in  weapon  system  acquisition. 


, *«4 

'  1  1 


1.  AFSC  Regulation  173-2,  2  October  1978, 
Confidence  Levels  of  Cost  Estimates  (Cost 
Analysis) 

2.  AFSC  Regulation  800-35,  31  January  1979, 
Independent  Schedule  Assessment  Program 
(Acquisition  Management) 

3.  The  Relationship  Between  Development  and 
Production  Hardware  Costs  in  Military  Weapons 
Systems,  September  1976,  Tecolote  Research, 
Inc. 

4.  Advanced  Strategic  Air  Launched  Missile 
(ASALM)  Independent  Cost  Study,  November  1979. 

5.  KC-135  Re-Engine  iCA,  May  1981 

6.  Long  Range  Combat  Aircraft  (B-1B)  ICA, 
September  1981 

7.  Next  Generation  Trainer  (NGT)  ICA,  October 
1982 

8.  IIR  MAVERICK  ICA,  February  1982  and 
February  1983 

9.  ASP  ECO  Model  Users  Guide,  March  1983, 
J.  Gibson,  Directorate  of  Cost  Analysis,  ASD, 
Wright-Patterson  AFB,  Ohio 

10.  C-5B  ICA,  April  1983 


"TEMPER  COST  FACTOR* 
EXAMPLE  OF  DERIVATION 


FIGURE  4 


TABLE  1 

COMPLEX  ITT  SCORE  CONFUTATION  WORKSHEET  AND  INSTRUCTIONS 
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ST  STEM  PROGRAM: 


1.  OEGREE  OF  CERTAIWTT  OF  DESIGN  COHFIGURATIOM 

!-I)  Very  Fine  (Chinese  Copy) 

0)  Typical 

I)  Modest  Uncertainty 

2)  Major  Uncertainty 

2.  RISK  OUE  TO  TECHNICAL  A0VAHCDg.1T: 


i-I)  Low 
0)  Routine 

I  Difficult 

2)  Complex 

3.  WETI-SERVICE  PARTICIPATION: 

(  0)  USAF  Only 

(  1)  Kjlt1-5arv1c« 


4.  MULTIPURPOSE  SYSTEMS: 

Tin — Tir - 

(  1)  Tes 

5.  CONTRACTOR  PROFICIENCY: 

!-l)  Better  Than  Average 

01  Typical  or  Average 

I  Less  than  Average 

2)  Significantly  Less  Hun  Avenge 

6.  OTHER  FACTORS 

(-1)  Positive  Impact  Reducing  Risk 

0)  No  Impact 

j  I)  Minor  Negative  Impact  Increasing  Risk 

j  2)  Medium  Negative  Impact  Increasing  Risk 
i  3)  Large  Negative  Impact  Increasing  Risk 
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TABLE  2 

UF  REQUIREMENT  PERCENT  PROGRAM  DATA 
NEW  AIRCRAFT  SYSTEMS 


3 


CGMPLE3ITY  SCORE  (5) 

2  CONCURRENCY 

(t) 

■ 

■ 

■ 

♦1 

♦2 

H 

m 

NORMAL  F50 
srnfni»i  r 

COMPRESSED 

F«in  V-HFW11  F 

20 

0 

I 

1 

m 

B 

m 

30 

15 

2 

1 

1 

1 

2 

2 
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25 

3 

1 

2 

2 

2 
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3 

3 

•  so 

35 

4 

2 

2 

2 

3 

4 

4 

60 

45 

5" 

3 

3 

3 

4 

4 

5 

70 

55 

6 

3 

4 

4 

5 

6 

6 

6 

80 

65 

7 

4 

4 
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7 

8 

90 

75 
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100 

85 
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7 

7 

9 

10 

12 

100 

10 

5 

6 

8 

9 

10 

11 

12 
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TABLE  3 


PROPOSAL  ANALYSIS 


CONTRACTOR  PROPOSAL 

CEILING  “ 


TARGET 


FURTHER  CONSIDERATIONS 


ROLL-OVER  RISK 


PAST  PERFORMANCE 


TABLE  A 

C-5B  INDEPENDENT  COST  ANALYSIS 

RANGk  OF  EXPECTED  COSTS 

(TV  DOLLARS  IN  f.llLUOUS) 


i  FY  84 
PRESIDENT’S 
BUDGET 


828R 


i 

*  —  Ha 


•All  costs  have  been  adjusted  and  do  not  reflect  actual  estimates. 
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•  All  costs  have  been  adjusted  and  do  not  reflect  actual  estimates. 
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WEAPON  SYSTEM  COST  RISK  REVIEWS 


DR.  JULES  J.  BELLASCHI 

DEPUTY  DIRECTOR,  PLANS  ANALYSIS  &  EVALUATION  (ARMY) 

There  is  always  a  considerable  cost  risk  associated  with  weapon  systems.  This  is  because  wea¬ 
pon  systems  have  life  cycles  which  often  run  more  than  20  years,  and  they  must  be  technology  and 
threat  reactive.  Therefore,  there  can  be  no  progress  without  taking  risks.  The  term  "life  cycle" 
suggests  a  process  with  a  beginning,  an  ending  and  a  series  of  phases  in  between.  Weapon  systems 
hdve  five  phases:  (1)  concept  exploration,  (2)  demonstration  and  validation  (Milestone  I),  (3) 
full  scale  development  (Milestone  II),  (4)  production  (Milestone  III),  (5)  deployment  which 
includes  disposal.  Of  greatest  concern  are  Milestones  I,  II,  and  III.  There  is  the  need  to  evalu¬ 
ate  the  cost  risk  at  each  of  these  milestones. 

The  Army  has  adopted  a  cost  risk  review  approach  as  systems  move  through  these  three  miletones. 
The  purpose  of  the  cost  risk  review  process  is  to  determine  the  cost  risk  associated  with  the  wea¬ 
pon  system  acquisition  programs  and  to  recommend  a  cost  position  to  the  Chairman  of  the  ASARC. 
There  is  a  general  officer  level  executive  committee  which  decides  which  weapon  system  programs 
will  be  subject  to  "cost  risk  review."  The  principle  criteria  will  be  those  systems  scheduled  for 
ASARC;  however,  other  systems  can  be  selected  on  an  "as  required"  basis.  The  second  tier  of  the 
cost  risk  review  structure  is  the  Cost  Risk  Review  Field  Team  which  reports  their  findings  to  the 
Executive  Committee.  The  Field  Team  is  kept  relatively  small  and  with  highly  experienced,  func¬ 
tionally  oriented  individuals. 

The  presentation  described  the  modus  operandi  of  the-Cost  Risk  Review  Field  Team,  and  in  gen¬ 
eral  terms,  case  results. 
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ABSTRACT 


"?)Thls  paper  presents  the  results  of  the  Army's 
effort  to  quantify  risk  and  to  budget  funds 
for  the  technological  uncertainty  in  the  pro¬ 
curement  of  weapons  systems.  This  effort  is 
known  as  Total  Risk  Assessing  Cost  Estimate 
for  Production  (TRACE -P).  This  paper 
describes  the  model  and  methodology  used  to 
quantify  this  uncertainty.  It  discusses 
implementation  of  the  system  and  the  results 
experienced  co  dace. 


BACKGROUND 


The  US  Army  Materiel  Development  and  Readiness 
Command  (DARCOM)  sets  challenging  goals  for 
the  development  and  procurement  of  major 
weapons  systems.  These  success-oriented 
programs  entail  risk  and  the  seemingly 
inevitable  occurrence  of  unfunded  contin¬ 
gencies.  It  is  no  easy  task  to  retain  our 
optimistic  funding,  schedule,  and  oerformance 
goals  while  presenting  a  fully  executable 
program  with  adequate  funding  for  tech¬ 
nological  risks.  Requests  for  funding  of 
contingencies  must  survive  the  budget  process. 
What  is  required  is  8  systematic,  organized, 
credible  approach  to  provide  realistic  cost 
estimates.  It  should  provide  assursnce  that 
funds  for  technological  risk  are  adequate  but 
not  excessive.  To  address  the  problem  the 
Army  developed  the  TRACE  concept. 


INITIATION  OF  THE  TRACE  CONCEPT 

The  original  concept  to  budget  for  the 
uncertainty  in  research  and  development  (R&D) 
waa  Initiated  by  the  Army  in  1974  under  the 
guidance  of  Mr.  Norman  A.  Augustine,  then 
Assistant  Secretary  of  the  Army  for  Research 
and  Development^- .  A  good  overview  of  this 
Total  Risk  Assessing  Cost  Estimate  (TRACE) 
concept  is  found  in  a  recent  paper  by 
LTC  John  D.  Edgar,  Defense  Systems  Management 
College^.  Extension  of  the  concept  to 


budgeting  for  the  technological  risk  in 
production  was  approved  on  30  April  1981  by 
then  Deputy  Secretary  of  Defense  Frank  C. 
Carlucci^.  This  extension,  known  as 
TRACE-P,  supports  Carlucci  initiatives  6  and 
11: 

Initiative  No.  6:  Budget  to  Most  Likely  Cost 
Initiative  No.  11:  Budget  Funds  for 
Technological  Risk 

Further,  on  22  July  1981,  the  Vice  Chief  of 
Staff,  Army,  stated  a  decision  to  implement 
the  concept  of  funding  to  cover  uncertainties 
in  production^.  The  Army  Cost  Discipline 
Advisory  Committee  monitors  the  progress  of 
this  and  other  high  level  initiatives. 

DEVELOPMENT  OF  TRACE-P 

Shortly  after  the  Carlucci  initiatives  were 
promulgated,  a  test  case  for  TRACE-P  was 
performed  on  the  Remotely  Piloted  Vehicle 
(RPV)  by  Ralph  Tate  at  the  US  Army 
AVRADCOM5.  In  June  1981,  the  US  Army 
Procurement  Research  Office  (APRO)  at  Fort 
Lee,  Virginia  was  directed  to  study  the 
applicability  of  this  concept  to  all  weapons 
systems  entering  production.  The  APRO 
report^  submitted  to  HQ,  DARCOM  identified 
the  sources  of  production  risks  during  the 
transition  into  production  and  grouped  them 
into  generalized  risk  categories.  APRO 
developed  the  theory  for  TRACE-P  and  refined 
the  analytical  techniques  used  to  quantify  and 
summarize  the  budgetary  Impact  of  risks. 

Based  on  the  APRO  report,  HQ,  DARCOM  refined 
Che  methodology  by  limiting  the  risks  to  the 
eight  specific  categories  shown  in  Figure  1. 
Eliminated  were  risk  areas  which  should  be 
corrected  by  good  management  or  by  other 
Carlucci  initiatives.  The  methodology  was 
verified  by  testing  the  model  on  the  M735 
round  using  historical  data.  The  results 
obtained  from  the  TRACE-P  model  approximated 
the  actual  coats  Incurred. 
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Analysis  of  M735 

Test  Results  (FY  82  $  In  Millions) 

BCE  $228.82 

BCE  +  Risk  282.50 

Actual  Cost  289.05 

In  February  1982,  the  TRACE-P  concept  was 
briefed  to  and  approved  by  the  CC,  DARCOM. 
Subsequently,  the  DARCOM  methodology  was 
briefed  to  HQDA  staff  principals  as  a 
recommended  approach. 

-  TRACE-P  CONCEPT 

TRACE-P  is  a  budget  policy  designed  to  improve 
the  capability  of  a  Project  Manager  (PM)  to 
minimize  the  impact  of  technological  risk  on 
his  program.  TRACE-P  provides  the  PM  with  a 
disciplined  method  of  costing  for  risk  as  well 
as  providing  higher  authorities  with  a  scien¬ 
tific  money  management  system.  TRACE-P 
identifies  and  promotes  understanding  of  the 
risks  Involved.  TRACE-P  aids  the  PM  in  coping 
with  those  risks  by  the  early  and  judicious 
application  of  funds.  These  funds  provide  an 
early  resolution  of  avoidable  risks  and  a 
funding  solution  for  unavoidable  risks. 

TRACE-P  is  used  tc  identify  and  budget  the 
funds  required  at  the  .5  probability  level 
(i.e.,  50/50  chance)  to  accommodate  the  known 
technological  risks  during  each  of  the  first 
three  years  of  significant  quantity  product¬ 
ion.  Three  distinct  actions  are  required  to 
quantify  risk  and  expand  the  use  of  budgeted 
funds  to  deal  with  uncertainty: 


1.  Identify  the  funds  required  for  risk, 

1. e.,  prepare  the  TRACE-P  estimate.  The  model 
used  is  described  below. 

2.  Budget  the  funds  required  for  risk.  The 
budgeting  process  for  TRACE-P  is  outlined 
below. 

3.  Manage  the  program.  A  separate  paper^ 
given  at  this  workshop  discusses  how  the 
Remotely  Piloted  Vehicle  (RPV)  Project  Manager 
(PM)  is  using  TRACE-P  to  manage  his  program. 

THE  MODEL 

Seven  basic  techniques  in  use  to  analyze 
acquisition  program  risk  are  described  in 
"Risk  Assessment  Techniques,  A  Handbook  For 
Program  Management  Personnel"®.  The 
TRACE-P  model  can  best  be  described  as  a  Work 
Breakdown  Structure  (WBS)  simulation.  The 
procedure  used  to  determine  the  funds  required 
for  technological  risk  during  procurement  of  a 
weapons  systems  is  as  follows: 

1.  Identify  the  major  subsystems  of  the 
weapons  system  using  the  WBS.  For  each 
subsystem,  identify  the  unfunded  technological 
risks  and  uncertainties.  Each  risk  is 
assigned  to  one  of  eight  risk  categories. 

These  categories  provide  the  framework  for 
applying  conventional  cost  estimating 
techniques  to  the  technological  risks  and 
uncertainties. 


FIGURE  1 


RISK  CATEGORIES 
INCLUDED  IN  TRACE-P 

•  THREAT  UNCERTAINTY 

•  MANAGEMENT 

•  MATERIALS/PURCHASED  PARTS 

•  FACILITIES/ EQUIPMENT 

•  LABOR 

•  DESIGN  CHANGES 

•  PRODUCIBII.ITY 

•  PERFORMANCE 


UNCERTAINTIES 
EXCLUDED  FROM  TRACE-P 

•  QUANTITY  CHANCES 

•  PERFORMANCE  IMPROVEMENTS 

•  POOR  MANAGEMENT 

•  INADEQUATE  FUNDING 

•  INFLATION 

•  CIVILIAN  PAY  ADJUSTMENTS 
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2.  Determine  a  cost  estimate  and  its 
distribution  for  each  contingency. 

Adjustments  are  made  to  the  data  if  necessary 
to  assure  that  the  technological  risks  are 
stochastically  independent. 


3,  Next,  the  probability  of  occurrence  of 
each  contingency  and  its  cost  distribution  are 
input  into  the  VERT  computer  program®. 

Using  a  Monte  Carlo  technique,  the  program 
derives  a  single  cost  distribution  for  all  the 
known  contingencies. 


4.  Apply  inflation  factors-and  display 
results . 


range  of  inputs.  A  strong  point  of 
Mr.  Alfieri's  model  is  the  analysis  of 
schedule  interactions  and  the  impact  of 
schedule  slips  on  the  total  cost  and  schedule. 
At  the  present  time,  the  output  in  terms  of 
technological  risk  cost  has  been  limited, 
considering  mostly  cost  estimating  uncertainty 
and  schedule  slippages.  While  it  is  more 
sophisticated  and  has  greater  potential  than 
the  current  TRACE -P  model,  it  requires  more 
detailed  knowledge  of  the  system  schedule  and 
imposes  data  requirements  on  the  contractor. 

By  comparison,  the  present  TRACE-P  model  is 
simple,  transportable,  and  demonstratable . 

Both  models  produce  an  audit  trail. 


5.  Include  Che  results  in  the  Baseline  Cost 
Estimate  (BCE)  and  Army  Materiel  Plan  (AMP)  by 
fiscal  year. 

TRACE-P  is  an  objective,  systematic, 
organized,  and  credible  approach.  It 
identifies  specific  technological  risks  and 
their  costs.  By  Including  these  risks,  it 
provides  'a  better  initial  cost  ea Cleats. 
TRACE-P  will  lessen  cost  growth  in  the 
investment  phase  of  weapons  systems  life 
cycle. 

One  other  risk  assessment  technique  that  shows 
significant  promise  for  deriving  TRACE-P 
estimates  is  described  in,  "Procedures  for 
Modeling  TRACE-P  Estimates",  by  Vincent 
Alfieri^0.  This  CECOM  approach  extends 
the  use  and  application  of  the  contractor’s 
Work  Breakdown  Structure  (WBS)  in  identifying 
risk  prone  areas,  and  combines  the  WBS  with 
probabilistic  networking  techniques  to  create 
a  data  structure  which  generates  risk  costs 
for  the  designated  program.  This  approach  has 
the  potential  to  consider  and  integrate  a  wide 


BUDGETING 

After  the  TRACE-P  estimate  is  prepared  and 
approved,  the  TRACE-P  value  is  to  be  included 
in  the  investment  portion  of  all  cost  and 
budget  estimates.  The  PM  submits  the  TRACE-P 
estimate  with  the  BCE  for  required  review  and 
approval.  He  also  enters  it  in  the  Army 
Materiel  Plan  (AMP).  The  Major  Subordinate 
Commands  (MSC)  and  HQ,  DARCOM  must  enter  the 
estimated  risk  cost  in  their  normal  programs 
and  budgets  within  the  Planning,  Programing, 
Budgeting  and  Execution  System  (PPBES).  The 
TRACE-P  estimate  becomes  part  of  the  budget 
requirements  and  must  survive  budget  cuts. 

The  budget  process  is  fraught  with 
difficulties.  Figure  2  illustrates  the 
barriers  encountered  by  the  initial  seven 
systems . 

STATUS  OF  TRACE-P 

In  March  1982,  the  Comptroller,  DARCOM  assumed 
responsibility  for  TRACE-P.  A  management 
concept  and  budgeting  methodology  were 


FIGURE  2 

STATUS  OF  TRACE-P  SYSTEMS 


SYSTEM 

TRACE-P 

COMMENTS 

XM-833 

FY  84  BUDGET 

POM  FOR  FY  85,  86 

AWAITING  CONGRESSIONAL  ACTION 

RPV 

POM  FOR  FY  86,  87,  S8 

RESULT  OF  ASARC 

PII 

- 

FY  84  $65M  SHORTFALL  ABSORBING  ALL 
FY  85  RESERVE 

AHIP 

- 

USOFA  CEILING 

S INCGARS -V 

- 

CUT  IN  PROGRAM 

STINGER/POST 

- 

CUT  IN  PROGRAM 

MCS 

- 

CUT  IN  PROGRAM 

developed.  Instructions  and  guidelines  were 
prepared  and  published  in  a  DARCOM  Letter  of 
Instruction11  on  TRACE-P  promulgated  on 
6  October  1982.  TRACE-P  requirements  were 
computed  for  the  XM-833,  the  Remotely  Piloted 
Vehicle  (RPV),  the  PERSHING  II,  the  Advanced 
Helicopter  Improvement  Program  (AHIP), 
SINCGARS-V,  the  Maneuver  Control  System  (MSC) , 
and  STINGER/POST.  A  summary  is  shown  in 
Figure  3.  The  budget  request  for  the  XM-833 
projectile  was  submitted  to  HQDA  on  25  May 
1982.  The  TRACE-P  equirements  for  the 
remaining  six  systems  were  submitted  to  HQDA 
on  1  April  1983.  HQDA  has  prepared  a  draft  of 
a  new  Army  Regulation  on  TRACE.  It  outlines 
the  TRACE  objectives,  policies,  responsi¬ 
bilities  and  procedures.  It  includes  both 
TRACE-P  and  TRACE-R  (RDTE) . 

The  FY  84  budget  contains  a  request  for 
TRACE-P  funds  for  the  XM-833.  The  Army  staff 
has  included  TRACE-P  funds  in  the  FY  1985-89 
POM  for  the  XM-833  (FY  1985-86)  and  the  RPV 
(FY  1986-88).  All  TRACE-P  fund  requests  will 
be  reviewed  during  the  Army  Materiel  Plan 
(AMP)  and  post  AMP  processes  prior  to  the  1985 


Office  of  the  Secretary  of  Defense  (OSD) 
submit  for  the  1985  budget.  HQ,  DARCOM's 
recommendations  will  be  used  to  determine 
which  weapons  system?  will  get  additional 
TRACE-P  funds  from  the  total  obligation 
authority  (TOA)  of  the  Army.  The  total  budget 
request  will  remain  the  same.  The  FY  84 
Budget  request  for  the  XM-833  has  been 
approved  by  the  House  Armed  Services  Committee 
(HASC)  and  the  Senate  Armed  Services  Committee 
(SASC) . 

The  House  Appropriations  Committee  (HAC) 
believes  that  there  is  a  need  for  management 
reserves  to  provide  the  necessary  flexibility 
to  address  unforeseen  circumstances.  The 
Committee  is  of  the  opinion  that  excessive 
management  reserves  should  be  avoided  and  if 
additional  funds  are  needed  for  a  particular 
development  effort,  specific  requests  should 
be  submitted  to  Congress.  If  it  should  become 
necessary  to  establish  management  reserves, 
the  Committee  expects  to  be  informed  of  the 
size  of  such  reserves,  the  project  involved 
and  the  reasons  for  the  establishment  of  such 
reserves . 


FIGURE  3 

RESULTS  TO  DATE 


SYSTEM 

INITIAL  3  YRS 

PROCUREMENT 

TRACE-P 

TRACE-P  AS 

1  OF  PROC 

XM-833 

270M 

25. 3M 

9.4Z 

AHIP 

1001 

135.2 

14.2 

RPV 

804 

111.2 

13.7 

PERSHING  II 

1235 

63.5 

5.2 

STINGER/POST 

1243 

58.4 

4.7 

MCS 

104 

11.6 

11.2 

SINCGARS-V 

446 

37.6 

8.4 
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PLENARY  PAPER 

THE  EQUITABLE  SHARING  OF  RISK  BETWEEN 
DEFENSE  AND  INDUSTRY 


THE  EQUITA8LE  SHARING  OF  RISK  BETWEEN 
DEFENSE  AND  INDUSTRY 


Jack  L.  Bowers 

CEO,  Sanders  Associates,  Inc. 

EQUITABLE  SHARING  OF  RISK 
AND  UNCERTAINTY 

Why  is  this  an  issue? 

•  There  is  a  Public  Record  of  Program  Failure 

®  There  is  a  history  of  attempted  solutions 

®  Regardless  of  Progress  Made— 

There  remains  a  perception  of  weakness  and 
need  for  improvement 


V-OJ-fl  210*001 

EQUITABLE  SHARING  OF  RISK? 


(Not  just  because  it  is  fair} 


With  intelligent  recognition  of  risk  Human  Beings  will 
perform  better 


Sharing  of  Risk  will  make  both  the  Buyer  and  the 
Seller  do  better 


V-00240-002 

115 


WHY  HOT  THE 
COMMERCIAL  WAY? 

In  commercial  endeavors  seller  assumes  risk 
Outstanding  successes! 

•  Automobiles 
®  Aircraft  Engines 
©  Computers 


Three  Answers: 

1. The  Military  Market  is  not  Predictable 

•  Changes  in  Threat 

•  Changes  in  DOD  Policy 

•  Changes  in  Congressional  Policy 

2.  Procurement  Regulations  do  not  allow  supplier 
to  recover  preincurred  costs 

3.  Advanced  State  of  the  Art 


V-t>«240>003»2 
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HISTORY  OF  RISK  IN 
MILITARY  ACQUISITION 


1950’S  CPFF  Era 

•  Technically  oriented 

•  Cost  atmosphere 

•  Many  successes 

•  Emerging  Problems 

1960’S  “Contract”  Era 

•  Shift  to  Fixed  Price,  Multiple  incentive  contracts 

•  Total  Package  Procurement 

•  C-5A 

•  Navy  Shipbuilding  claims  (into  70’s) 


V-U-02 *0-004-1 

Lesson  Learned: 

Contract  form  alone  is  not  the  Answer 
Conclusion: 

We  need  tools  to  help  the  manager  control  the 
process 

Not  a  Process  to  control  the  Manager 


V-M-02*0-00*-Jl 
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HISTORY  OF  RISK  IN 

MILITARY  ACQUISITION 

The  Management  Era: 

1970  Packard 
DOD  instruction  5000.1 
DOD  Instruction  7000.2 

1980— Carlucci  Initiatives 


V-*X>2ftO>  004-3 

risk  in  military  acquisition— 

THE  FUTURE 

Thesis: 

If  We: 

1.  Follow  guidance  of  Packard 

2.  Implement  Carlucci  initiatives 

3.  Improve  Planning  and  Budgeting 

4.  Train  Industry  and  Military  Managers 

We  Will: 

1.  Improve  Procurement  Process 

2.  Learn  to  Manage  Risk 

3.  Establish  basis  for  equitable  sharing  of  risk 

ft 

V4)4UM0SA 
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WHAT  ARE  THE  SOURCES  OF 
RISK  AMD  UNCERTAINTY? 

1.  Advanced  technology 

2.  Attaining  satisfactory  Reliability  and  Quality 

3.  Integration  of  complex  systems 

4.  Availability  of  manpower 

5.  Availability  of  facilities 

6.  Instability  of  Programs 

a.  Cancellation 

b.  Change  in  schedule 

c.  Change  in  requirements  (threat) 

7.  Inadequate  Budgeting 

V-4XJ2A0-004 

“BUDGETING  TO  MOST 

LIKELY  COST” 

(NAVAL  AVIATION  INDUSTRIAL  COUNCIL  STUDY) 

Causes  of  Unrealistic  Pricing  during  Advocacy  Phase 

•  Low  Defense  Budgets  following  VietNam 

•  Micromanagement— program  instability 

•  Understated  Inflation 

•  Production  of  too  many  systems 

•  Noneconomic  production  rates 

v-*x«»a-oo7 
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KINDS  OF  ACTIONS  CONSIDERED 
IN  DEALING  WITH  RISK 


•  Identifying  Risk  and  Uncertainty 

•  Minimizing  Risk  by  Careful  Planning 

•  Acceptance  of  Individual  Risks 

•  Risk  Sharing  Plan  at  Contract  Inception 

•  Managing  Risk  sharing  during  Contract 

•  Long  Range  Actions  to  improve  Risk  Environment 


V-U-4240-001 

ACTIONS  TO  LIMIT  AND 
EQUITABLY  SHARE  RISK 

•  Prepare  sound  Procurement  Plan 

•  Review  Plan  with  Industry  (Modify) 

•  Select  Proper  form  of  Contract 

•  Increase  use  of  Award  Fee  Contracts 

•  Increase  use  of  Multiyear  Contracts 


V4W2IM0M 
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ACTIONS  TO  LIMIT  AND 
EQUITABLY  SHARE  RISK  (CONT) 

•  Establish  Capital  Investment  Incentives 

•  Eliminate  multiple  Best  and  Final  Offers 

•  Increase  use  of  VECP’s 

•  Make  Cost/Performance  Tradeoffs 

•  Improve  Military/Contractor  Business  Relationship 


V-U-02l0>00t-2 

PREPARE  SOUND 
PROCUREMENT  PLAN  (ASPR) 

•  Performance  Requirements 

•  Reliability,  Quality  and  Maintainability 

•  Evaluation  of  Risk 

•  R&D  Plan 

•  Test  and  Evaluation  Plan 

•  Production  Plan 

•  Schedules 

•  ILS  Plan 

•  Estimated  Cost 


REVIEW  PLAN  WITH  INDUSTRY 

Review  Plan 

•  Technical 

•  Schedule 

•  Contract  Form 

Modify  Plan 

•  Specifications 

•  Schedule 

•  Contract 

•  Estimate  of  most  likely  cost 

V-IJ-O2W-011 

SELECT  PROPER  FORM 
OF  CONTRACT 

•  DOD  Instruction  5000.1  states  that  Contract  Form 
should  be  consistent  with  risk 

•  Use  simple— straightforward  contract  forms 

•  Contract  Form  should  not  be  used  for  negotiating 
tool 

•  Strict  Contract  forms  should  not  be  used  to 
eliminate  inexperienced  contractors 


V-4>42I041)-1 
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SELECT  PROPER  FORM 
OF  CONTRACT  (CONT) 

“Contract  Types  inconsistent  with  inherent  technical 
risks,  stage  of  production,  nature  of  task,  or  which  do 
not  recognize  economic  reality  are  not  in  best 
interest  of  government  or  contractor” 


V-U-C2I0-012-2 

iNCREASE  USE  OF  AWARD 
FEE  CONTRACTS 

•  Profit  Incentive  frequently  used 

•  Award  fee  adds  performance  review 

•  Pride  of  Contractor  personnel  becomes  involved 

•  Responsibility  of  Military  P/M  is  emphasized 

•  Risk  sharing  is  negotiated  on  real  time  basis 


V4MU»4tl 
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INCREASE  USE  OF 

MULTIYEAR  CONTRACTS 


•  Military  Receives  Lowest  price  Low  Risk 
.  industry  Gains  Stability  (Otfsetting  Price  Risk) 


But: 

•  Loses  Budget  Flexibility-Bad? 

•  inhibits  Technical  Change— Bad? 


V.U-0210-014 

ESTABLISH  CAPITAL 
INVESTMENT  INCENTIVES 

•  Productivity  improvement  needed 

•  Capital  Investment  expensive 

•  Stability  of  Future  Government  Procurement 

•  Contractor  Investment  Risk  should  be  balanced 
with  Profit  incentive 

•  Exploratory  work  in  process 
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ELIMINATE  MULTIPLE  BEST  AND  ! 

FINAL  OFFERS  j 

•  Multiple  Best  and  Final  Offers  should  be  eliminated 
(Contractors  are  encouraged  to  submit  unrealistic 
offers) 

•  Justification  should  be  demanded  of  any  cost 
change 

•  Cost  changes  should  be  allowed  only  in  areas  of 
competitive  discussion 

•  Risk  will  be  reduced— both  parties 


V-M-C2IG-01H 


“The  Government’s  interest  is  best  served  by  a 
negotiation  to  the  lowest  attainable  cost  by  a  viable 
contractor,  not  in  driving  the  earnings  of  the  selected 
contractor  to  the  lowest  level” 


V-IMitMII-l 
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INCREASE  USE  OF  VECPs 


VECPs  are  not  clearly  understood  or  used  effectively 
Contractor  and  Military  risk  will  benefit  thru  wider  use 


V43-O2JO-017 


MAKE  COST/PERFORMANCE 
TRADEOFFS 

Unknown  factors  usually  drive  cost  up 
Performance  requirements  are  jsually  rigid 
Tradeoffs  will  reduce  and  balance  risk 


V-U-3M0-4U 
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IMPROVE  MILITARY- 
CONTRACTOR  BUSINESS 
RELATIONSHIP 


Nonadversarial 

Program  Managers  should  work  together  from 
same  data 

Both  loyal  to  the  program 

Each  loyal  to  their  own  organization 

The  contract  should  be  written  to  resolve  conflict 

Only  good  management  can  produce  equitable 
sharing  of  risk 
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APPROXIMATELY  BOUNDED  RISK  REGIONS 


J.  J.  Wilder  &  R.  L.  Black 

Grumman  Aerospace  Corporation 
Bethpage,  New  York  11714 


ABSTRACT 

In  conventional  WBS-oriented  costing,  each  WBS  ele¬ 
ment  cost  is  estimated,  and  the  total  cost  is  taken  to  be 
the  sum  of  the  element  costs.  In  assessing  the  overall 
uncertainty,  the  element  cost  is  treated  as  a  random 
variable,  and  the  characteristics  of  the  random  variable 
estimated.  The  overall  cost  is  then  expressed  as  another 
random  variable  by  appropriate  summing  of  the  ele¬ 
ment  variables. 

The  conventional  analytical  approach  is  to  assume  that 
the  element  costs  are  independent.  Under  this  assump¬ 
tion,  the  additive  (A)  moments  of  the  element  random 
variables  are  determined,  and  added  to  determine  the  A 
moments  of  the  sum.  A  curve  is  fitted  to  the  sum’s  A 
moments  to  represent  the  cumulative  distribution  func¬ 
tion  of  the  total  cost. 

We  now  make  the  alternative  assumption  that  the  ele¬ 
ments  are  linearly  dependent,  and  develop  a  set  of  D 
moments  that  have  the  same  additive  properties  of  the 
A  moments  under  the  independence  assumption.  In  sim¬ 
ilar  fashion  to  the  A  moment  procedure,  element  D  mo¬ 
ments  are  determined  and  added  to  yield  the  D  mo¬ 
ments  of  the  total  cost.  A  curve  is  fitted  to  these  mo¬ 
ments  to  represent  the  cumulative  distribution  function 
of  the  total  cost.  These  two  curves  may  be  considered 
the  boundaries  of  a  probability  risk  region. 

Working  on  risk  analysis  with  project  management  per¬ 
sonnel  has  shown  the  need  for  a  quick,  easily  imple¬ 
mented  method  tc  calculate  this  risk  region.  Based  on 
these  needs  programs  have  been  developed  that  are 
suitable  for  use  on  a  handheld  calculator.  For  conven¬ 
ience  they  are  presently  programmed  for  the  TI-59. 

Like  the  methodology  discussed  previously,  they  use 
moments  to  approximate  distributions.  Procedures  have 
been  developed  for  both  the  independent  and  dependent 
cases. 

INTRODUCTION 

A  Grass  Root  estimate  consists  of  dividing  a  complete 
project  into  identifiable  activities  which  taken  together 
include  ail  required  tasks  —  a  Work  Breakdown  Struc¬ 


ture  (WBS).  The  cost  to  complete  each  activity  is  sepa¬ 
rately  estimated,  usually  based  on  a  mixture  of  experi¬ 
ence  and  analogy  with  previously  completed  similar 
tasks.  These  WBS  element  estimates  are  then  added  to 
become  a  total  cost  estimate.  Usually  manhours  are  es¬ 
timated  and  dollars  are  found  by  applying  the  rates  ap¬ 
plicable  to  each  skill  code.  Also  various  additional  fac¬ 
tors  are  also  included,  such  as  indirect  allocations, 

G&A  and  fee.  These  factors  add  some  computing  com¬ 
plication  but  do  not  change  the  basic  approach,  so  they 
will  be  ignored  here. 

An  uncertainty  analysis  covers  the  same  ground.  It  uses 
the  same  WBS.  However,  the  cost  estimates  of  the  in¬ 
dividual  WBS  elements  are  expressed  as  probability 
distributions  rather  than  point  estimates.  Then  it  adds 
the  distributions  to  obtain  a  total  probability  function. 

This  resulting  function  expresses  the  range  of  possible 
,  costs  in  terms  of  their  probabilities  of  being  achieved.  rC. 

INPUT  DENSITY  FUNCTIONS 

Since  we  have  stated  that  our  estimates  are  to  be  ex¬ 
pressed  as  probability  distributions  rather  than  point 
estimates,  it  is  necessary  to  determine  what  form  our 
input  probability  density  function  should  take.  If  we 
assume  that  the  project  is  technically  feasible,  then  for 
each  element: 

•  There  is  a  greatest  lower  bound  for  the  resources 
required  for  the  task  to  be  accomplished  with  a 
probability  of  zero 

•  There  is  a  least  upper  bound  for  the  resources  re¬ 
quired  for  the  task  to  be  accomplished  with  a 
probability  of  one. 

The  actual  shape  of  the  density  function  is,  in  reality, 
unknown,  and  probably  unknowable.  We  can,  however, 
assign  characteristics  to  it  that  would  be  logical. 

Lit  should  have  fixed,  positive  upper  and  lower 
bounds 

2.  It  should  not  be  necessarily  symmetric 

3.  It  should  be  unimodal 

4.  It  should  be  computationally  simple. 


Many  density  functions  meet  all  or  most  of  these  cri¬ 
teria.  Perhaps  the  most  widely  used  is  the  Beta  distri¬ 
bution  suggested  by  the  pioneering  work  of  Dieneman 
(3),  and  used  by  Klein  (5,  6)  and  many  others. 
McNichols  (8)  uses  a  Weibul  distribution,  which  does 
not  require  the  strict  feasibility  assumption  since  it  is 
open-ended  at  the  top.  In  this  paper,  we  will  use  the 
.  triangular  distribution.  It  is  completely  characterized  by 
three  points,  low  (/)  with  an  associated  probability  of 
zero,  mode  ( m )  the  most  likely,  or  modal  value,  and 
high  ( h )  with  an  associated  probability  of  one.  We 
usually  assign  the  modal  value  t '  the  nominal  or  point 
estimate.  This  distribution  meets  the  criteria  mentioned 
above,  and  a  very  useful  characteristic  is  that  its  in¬ 
verse  transform  is  closed  form  and  quite  simple,  mak¬ 
ing  it  very  convenient  for  Monte  Carlo  or  other 
simulations. 

The  requirement,  then,  is  for  three  estimates  for  each 
element;  /,  m,  and  h.  In  some  cases  the  analyst  may 
feel  uncomfortable  with  trying  to  estimate  the  zero  and 
one  probability  points.  In  this  case,  we  ask  for  a  low 
estimate  (l1)  and  an  associated  probability  of  underrun 
(s),  the  mode  (m)  and  a  high  estimate  (hr)  and  a  prob¬ 
ability  of  overrun  (p ).  The  /  and  h  values  are  calculated 
easily  as  shown  in  Reference  (12). 

In  some  instances  it  may  only  be  possible  to  assign 
bounds  to  the  estimate.  In  that  case,  we  use  the  uni¬ 
form  distribution.  The  characteristics  of  the  triangular 
and  uniform  distributions  are  described  in  detail  in 
References  (9)  and  (10). 

INDEPENDENT  ELEMENTS 

If  it  can  be  assumed  that  these  estimates,  or  WBS 
elements  are  independent,  then  our  normal  approach  is 
to  use  the  method  of  moments.  This  approach  follows 
the  general  approach  of  McNichols  (7)  and  is  described 
in  detail  in  References  (9),  (10),  and  (11).  For  this 
reason,  only  a  brief  summary  is  included. 

If  x  is  a  random  variable  with  mean  p  (first  origin  mo¬ 
ment)  and  using  the  nomenclature  to  signify  its  i1* 
central  moment,  then  its  additive  moments  At  are  as 
follows: 

A,  = 

A-,  —  /iri) 

A]  =  M<3) 

Aa  =  -  3  0 r(2))2 

The  useful  property  of  these  A  moments  is  that,  for  all 
independent  x(’s,  the  A  moments  of  the  sura  of  the  x/s 
is  the  sum  of  the  A  moments  of  the  individual  x,’s. 

To  perform  the  analysis,  we  determine  the  four  A 
moments  of  each  WBS  element  and  add  them  to  deter¬ 
mine  the  output  distribution’s  A  moments.  We  then  fit 


a  Beta  distribution  to  these  four  moments  [Reference 
(1)]  and  assume  that  it  represents  the  output  density 
function.  We  have  developed  a  library  of  FORTRAN 
subroutines  to  determine  these  for  triangular  and  uni¬ 
form  distributions  and  for  the  Beta  fit. 

A  main  program  is  written  to  input  the  data,  call  out 
the  required  subroutines,  and  supply  required  format¬ 
ting.  These  programs  are  also  written  in  BASIC  for  the 
HP  9830. 

Use  of  the  four  additive  moments  enables  us  to  shape 
the  output  density  function,  since  a  function  of  A4  and 
A:  defines  the  kurtosis,  or  peakedness,  and  a  function 
at  A3  and  A:  defines  the  skewness.  The  output  Cumula¬ 
tive  Distribution  Function  (CDF)  is  easily  found  by  a 
simple  numerical  integration,  and  the  output  probability 
statements  obtained. 

By  examining  the  element  results,  useful  insights  may 
be  obtained  into  the  risk  elements  of  the  project.  The 
risk  drivers  are  those  elements  with  the  greatest  vari¬ 
ance,  while  the  cost  drivers  are  those  with  the  greatest 
means.  They  are  not  necessarily  the  same. 

DEPENDENT  ELEMENTS 

The  assumption  of  complete  independence  among  proj¬ 
ect  elements  is  often  troublesome.  Perhaps  its  wide  use 
is  due  more  to  ease  of  analytical  or  simulation  solutions 
than  to  practical  applicability.  In  most  cases,  it  is  not 
intuitively  pleasing  to  assume  complete  independence, 
since  poor  results  in  one  project  element  do  tend  to  rip¬ 
ple  through  related  elements.  The  independence  as¬ 
sumption,  due  to  operation  of  the  Central  Limit 
Theories  will  likely  have  a  relatively  steep  CDF,  hence 
may  tend  to  understate  the  variability  or  risk.  This  rep¬ 
resents  an  optimistic  view  of  the  possible  outcomes.  It 
is  our  opinion  that  the  assumption  of  independence  is 
seldom  completely  valid. 

If  we  make  the  opposite  assumption,  i.e. ,  that  there  is 
complete  linear  dependence  amoug  the  project  ele¬ 
ments,  in  effect  we  say  that  any  problem  with  any  ele¬ 
ment  will  be  reflected  in  all  elements,  and  conversely 
any  “good  luck”  will  be  similarly  reflected.  This  as¬ 
sumption  may  not  be  valid  in  many  situations,  but  we 
feel  that  it  is  closer  to  reality  than  the  independency 
assumption,  and  the  region  between  the  two  assump¬ 
tions  may  be  considered  to  bound  the  set  of  inter¬ 
mediate  outcomes. 

Under  the  assumption  of  complete  linear  dependence,  a 
set  of  "D”  moments  is  developed  that  have  the  same 
additive  characteristics  as  the  A  moments  under  the  in¬ 
dependence  assumption.  Thus,  the  k^D  moment  of  the 
sum  of  n  dependent  variables  is  the  sum  of  the  fr*  D 
moments  of  the  n  elements. 


As  shown  in  Appendix  A,  the  k*  D  moment  is  the 
root  of  the  k *  central  moment,  Mill.  The  first  O  mo¬ 
ment  is  the  first  origin  moment,  or  mean,  and  is  there¬ 
fore  equivalent  to  the  first  A  and  M  moments. 

As  shown  in  Appendix  A,  there  is  a  simple  relationship 
between  the  various  moments.  Using  the  following 
notation, 

dependent  additive  moment 
independent  additive  moment 
central  moment  (Mill)' 

C,  =  D,  =  n 
C2  =  DJ  =  a1 
C,  =  Z>,J 
C*  -  3C22 
C4  =  Df 


Dk  = 

Ak  = 

Q  = 
— 

a2  = 
A]  = 
A4  = 


For  an  analytical  solution,  the  existing  library  of  FOR¬ 
TRAN  subroutines  was  modified  to  solve  for  the  addi¬ 
tive  D  moments.  Curve  fitting  of  the  Beta  distribution 
uses  the  original  routines  after  making  the  D  to  A 
transformation  at  the  summary  level. 


To  illustrate  these  techniques,  a  hypothetical  project  is 
shown  with  six  additive  elements.  Input  variables  were 
all  triangular  distributed;  three  transformed  by  Cost 
Estimating  relationships  of  the  form  y  =  a  +  bxr,  and 
three  unchanged  by  a  CER,  i.e.,  a  =  0,  b  =  n  =  1. 
The  input  data  are  shown  in  Table  1.  The  example  was 
solved  for  the  independent  case  by  our  standard  method 
of  moments  technique.  Reference  (9).  For  the  depen¬ 
dent  case,  both,  the  analytical  method  of  D  moments, 
and  a  100,000  iteration  simulation  were  used.  The  sim¬ 
ulation  used  the  “SLICE”  technique  described  in 
Reference  (2)  with  calculated  means. 


ELEMENT 

LOW 

MODE 

HIGH 

• 

b 

n 

1 

9 

.2 

16 

14 

6 

0.86 

2 

14 

18 

28 

23 

8 

0J1 

3 

8 

9 

14 

17 

2 

1 2 

4 

31 

38 

55 

0 

1 

1 

5 

27 

29 

37 

0 

> 

1 

6 

40 

45 

54 

0 

1 

1 

10*3-10 

Table  1  Example  Input  Data 


The  results  are  tabulated  in  Table  2.  The  moments 
shown  in  the  table  are  the  conventional  A  moments  for 
ease  of  comparison.  In  the  dependency  cases  these 
were  found  by  transforming  the  summary  D  moments 
using  the  equation  described  above.  The  values  of  the 
exponents  or,  0  of  the  fitted  Beta  distribution  are  also 
shown  as  well  as  their  end  pou  s  and  ranges. 
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DEPENDENT  CASE 

INDEPENDENT  CASE 

TECHNIQUE/ 

PARAMETERS 

SIMULATION 

ANALYTICAL 

ANALYTICAL 

A1 

2.777E2 

2.777E2 

2.777E2 

A2 

4.459E2 

4A73E2 

9.168E1 

A3 

3.Z14E3 

2-833  E3 

1.437E2 

A4 

-1.224E5 

-1 .182E5 

-1 J41E3 

SKEWNESS 

0.341 

0J00 

0.164 

KURTOSIS 

2J84 

2.409 

2352 

0.841 

1.142  * 

10.284 

0 

1 .968 

2J13 

16379 

LOW 

237.6 

234.5 

2355 

RANGE 

104.7 

110.0 

1083 

HIGH 

342J 

3445 

3435 

Table  2  Example  Results 


It  can  be  seen  by  inspecting  the  table  that  the  simula¬ 
tion  results  are  a  very  close  approximation  of  the  ana¬ 
lytical  approach  using  the  additive  D  moments.  As  ex¬ 
pected,  the  independent  case  has  a  smaller  variance,  is 
less  positively  skewed,  and  its  kurtosis  is  close  to  that 
of  a  normal  distribution. 

In  Figure  1,  the  CDFs  for  the  analytical  and  simulation 
dependent  approaches  are  plotted.  It  can  be  noted  that 
the  curves  for  dependent  simulation  and  analytical  ap¬ 
proaches  are  indistinguishable. 


104  3-30 

Fig.  1  Cumulative  Distribution  Functions: 
Simulation  &  Analytical 


Figure  2  illustrates  the  bounded  region  within  the 
curves  of  the  dependent  and  the  independent  cases.  It  is 
a  matter  of  judgement  to  determine  where  in  this  area 
the  “truth”  lies.  The  author’s  preference  is  to  use  the 
dependent  case. 


VS:  .T  •  /-, 
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1043-40 

Fig.  2  Cumulative  Distribution  Functions 


“EQUIVALENT  TRIANGLE  APPROXIMATIONS” 

It  has  been  noted  empirically  that  the  kurtosis  of  a  tri¬ 
angular  distribution  is  2.4,  and  a  proof  is  given  in  Ap¬ 
pendix  B.  It  was  also  noticed  that  the  kurtosis  of  the 
sum  of  dependent  triangular  distributions  is  also  2.4.  A 
proof  of  this  is  offered  in  Appendix  C. 

These  relationships  indicate  that  a  good  approximation 
for  the  sum  of  dependent  triangular  distribution  might 
be  a  triangular  distribution  having  the  mean,  variance, 
and  skewness  of  the  calculated  summary  dependent 
moments. 

Following  the  procedure  outlined  in  Appendix  D,  the 
“equivalent”  triangular  model  for  the  illustrative  exam¬ 
ple  is  l  =  232.32,  m  =  266.64,  h  =  334.14.  The  re¬ 
sults  of  the  simple  area  determination  to  calculate 
points  on  the  CDF  curve  (Figure  3)  are  shown  by  the 
“X”  values  for  this  dependent  curve,  plotted  on  the 
analytically  determined  CDF.  The  density  functions 


Fig.  3  Cumulative  Distribution  Functions 


derived  analytically  and  by  simulation  are  shown  as 
Figure  4,  as  well  as  the  “equivalent”  triangle. 


1043-6G 

Fig.  4  Probability  Density  Functions 


For  the  case  of  summed  triangular  distributions,  the  en¬ 
tire  analysis  may  be  done  quickly  using  programs  writ¬ 
ten  for  the  TI-59  handheld  calculator.  Programs  are 
available  for  calculating  moments  of  triangular  distribu¬ 
tions  (A,  M,  and  D)  and  for  the  area  calculation  for  de¬ 
termining  points  on  the  CDF  to  determine  confidence 
limits  for  the  dependent  case.  A  good  approximation 
for  the  independent  case  is  to  add  the  means  and  vari¬ 
ances,  and  assume  normality  per  Central  Limit  Theo¬ 
rem  (as  long  as  a  reasonable  number  of  roughly  equal 
estimates  are  being  considered). 

CONCLUSIONS 

The  extension  of  the  method  of  moments  to  include  ad¬ 
ditive  D  moments  offers  a  relatively  simple  and 
straightforward  solution  to  determining  the  cost  risk 
associated  with  a  WBS-oriented  program  when  the  in¬ 
dependence  assumption  cannot,  or  should  hot  be  made. 
The  solution  is,  we  think,  bounded  by  the  CDF  repre¬ 
senting  the  independence  assumption,  an  optimistic  ap¬ 
proach,  and  the  CDF  represented  by  the  complete  lin¬ 
ear  dependence  assumption,  a  pessimistic  approach. 

The  methodology  is  simple  enough  so  that  most  anal¬ 
yses  may  be  performed  by  a  programmable  handheld 
calculator,  such  as  the  TI-59,  or  when  CER  transforms 
exist,  a  desk-top  computer  (HP9830  or  HP9845)  will 
usually  suffice. 

For  large  problems,  a  library  of  FORTRAN  subrou¬ 
tines  has  been  prepared,  and  these  may  be  assembled 
by  a  suitable  main  program. 
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APPENDIX  A* 

DERIVATION  OF  "D”  MOMENTS 

mk(x)  =  E(x-pxy  mt(y)  =  Eiy-pjf 

If  x  and  y  are  linearly  dependent,  then 

y  =  A  +  Bx,  and  Py  =  A  +  Bpx 

(x~Px)  +  (y-Py)  =  (.x-pj  +  {A  +  Bx-  A  -  Bpj) 

~  (x-Px)  +  B(x-Px) 

=  (1  +B)(.x-px)  (1) 

mk(y)  =  Eiy-pyf  =  E  (A  +  Bx  -  A  -  Bxff 
=  Bt  (x-pxY 
=  5*  mt  {x) 


•Derivation  suggested  by  Dr.  R.  Ddwd,  Operations 
Analysis  Technical  Staff 


\Jmk(y)  =  Byjmk( x)  (2) 

mt(jr+y)  =  E((x-px)  +  (y-py))k 

=  £((1+B)  (x-pjf  from  (1) 

=  (i  +By  e  (x-pxy 

=  (l+fl)*  mk(x)  by  definition 

=  ((1  +B)^mk(xY 
=  ( vm,  {x)  +  B  \Jmk 

mk(x+y)  =  (  y/mk  (x)  +  from  (2) 

then 

y/mk  (x+y)  =  ^mk  (x)  +  yfmk  (y) 

It  is  clear  by  induction  that 

\Jmk'£lxi  =  £  -t/mk(x) 


We  will  use  the  following  notation: 

yjmk  (x)  =  Dk  for  dependent  additive  moments 

Ak  for  independent  additive  moments 
CK  fvr  *.■"  Central  Moment  (jfeDl) 

Ci  =  p 

D,  =  p 

then 

A\  =  Ci  =  Di  —  p 
A2  =  C2  «  Df  =  a1 
Aj  =  C3  =  Dj3 
At  =  Ct  -  3 Cf 
C,  =  Df 


APPENDIX  B 

DERIVATION  OF  THE  KURTOSIS  OF  A 
TRIANGULAR  DISTRIBUTION 


r 

K  =  — L-  (CK  is  the  K!h  Central  Moment) 

C2 


Consider  the  A  (0,  m,  1) 

£[x‘]  = 

1  +  m  +  m1  +  m3  +  m* 

15 

(1)  Ref  (9) 

E[x>]  = 

1  +  m  +  m1  +  m3 

10 

(2) 

£[*2]  = 

1  +  m  +  m2 

6 

(3) 

E[x]  = 

1  +  m 

3 

(4) 

C2  =  E{XL}  -  £[X]2 


might  be  a  triangle  with  skewness,  variance,  and  mean 
corresponding  to  the  summed  D  moments. 


1  -  m  +  m1 


Substituting  (3)  and  (4) 

„ ,  1-2 m  +  3  m1  -  2m 3  +  m 4 

then  C22  =  - — - 


C4  =  E[x*]  -  4 EW\E[x\  +  6E[x1]E[x]1  -  3£[r]4 
Substituting  (1),  (2),  (3),  (4) 

r  _  1  —  2m  +  3 m2  -  2m3  +  m 4 

2  _  _ 

'hen  K  =  -£_=i?±  =  2.4 
C22  135 


We  first  determine  a  value  m,  the  mode  of  a 
A  (0,  m,  1). 

From  Reference  (9)  it  is  shown  that  for  the  xA 

(0,  m,  1) 

.  2 m3  —  3m2  —  3m+2 


m2  -  m+1 


Coefficient  of  Skewness 


APPENDIX  C 

KURTOSIS  OF  THE  SUM  OF  DEPENDENCY 
TRIANGULAR  DISTRIBUTIONS 


(t) 


=  2.4  for  any  A  distribution 
as  shown  in  Appendix  B 


KD2*  =  Dt* 

K*  D2  =  D4 
then  summing: 

K*  L  Dv  =  L  Dm 

in  =  ^ 


K  =  /  E  Dt<\ 

(r  dJ 

Thus,  the  kurtosis  of  the  sum  of  dependent  triangles  is 
shown  to  be  2.4. 


The  TI-59  was  programmed  to  find  7,  from  a  manually 
entered  value  of  m,  and  an  iterative  search  performed 
to  find  a  value  of  m  corresponding  to  the  desired  7,. 

(A  more  elegant  search  routine  will  be  written  as  time 
permits.) 

With  this  value  of  m,  we  have  a  A  (0,  m,  1)  with  the 
desired  skewness.  We  now  want  to  find  a  similar  trian¬ 
gle  (0,  km,  k)  with  the  required  variance  if  K[A]  = 
required  variance,  and  V[.]  is  the  variance  of  the 
A  (0,  m,  1)*,  then 


_  W] 

V  V[.\ 


and  our  A  with  the  required  variance  is 
(0,  km,  k) 

•n.i-  1~m+ml 
18 

We  now  need  to  shift  the  triangle  along  the  X  axis  to 
give  us  a  triangle  L,  L+krn,  L+k,  and  all  we  need  now 
is  the  value  of  L: 


L+M+H 


definition 


L+L+km+L+k 


APPENDIX  D 

DETERMINATION  OF  THE  "EQUIVALENT” 
TRIANGLE 

Since  the  kurtosis  of  the  distribution  resulting  from  the 
addition  of  dependent  triangular  distributions  is  always 
2.4,  this  suggests  that  a  good  model  of  this  distribution 


solving  for  L: 
L  =  m  “ 


k(m+  1) 
3 


and  our  "equivalent”  triangle  is 
L,  L+km,  L+k 
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possess  an  inherent  variability,  then  the 
resulting  aggregated  cost  of  the  systems 


I.  INTRODUCTION 

A.  BACKGROUND 

Virtually  all  cost  estimates  -  whether 
for  private,  commercial,  or  governmental 
systems  -  are  presented  as  fixed  point  esti¬ 
mates  without  qualification,  and  are  conmon- 
ly  assumed  to  possess  a  high  level  of 
certainty.  However,  even  a  cursory  examin¬ 
ation  of  the  nature  of  costs  reveals  that 
they  are  intrinsically  highly  variable. 
Unless  the  cost  of  a  commodity  or  system 
element  is  artificially  .fixed,  it  will  vary 
locally,  regionally,  nationally  and,  to 
compound  the  variability,  temporally.  Raw 
material  costs,  labor  costs,  utility  costs 
all  vary,  if  not  locally,  then  regionally 
and  nationally.*  Complex  systems  are 
essentially  the  results  of  labor  (manage¬ 
rial,  engineering,  manufacturing,  etc.), 
equipment,  and  energy  applied  to  raw 
materials.  If  the  costs  of  these  elements 

•* 


created  would  likewise  display  a  correspond¬ 
ing  variability.  The  foregoing  sources  of 
cost  uncertainty  are  further  compounded  by 
uncertainties  inherent  in  both  the  specific 
details  of  the  system  being  costed,  and  in 
the  amounts  of  labor  and  materials  that  will 
be  required. 

Regardless  of  what  technique  may  be  used 
to  generate  system  cost  estimates,  the  vari¬ 
ability  inherent  in  the  cost  of  the  system 
elements,  when  combined  with  the  intrinsic 
uncertainty  in  the  cost-impacting  details  of 
the  system  together  with  such  factors  as 
inflation  impacts,  should  make  any  unqual¬ 
ified  point  cost  estimate  highly  suspect.  A 
cost  estimate  at  best  is  only  an  approximate 
representation  of  the  expected  system  cost 
rather  than  the  precise  price  that  it  is  too 
commonly  assumed  to  be. 


*  According  to  the  1980  Dodge  Manual,  published  by  McGraw  Hill,  the  wage  rates  for  laborers  in 
Raleigh,  N.C.  was  $6.06  per  hour,  whereas  430  air  miles  away  in  Cleveland,  Ohio,  it  was 
$17.79  per  hour  or  190Z  greater.  Salaries  of  professional  engineers  likewise  show  consider¬ 
able  variability  as  reported  in  Professional  Engineer  Income  and  Salary  -  1981,  published  by 
the  National  Society  of  Professional  Engineers.  Based  on  samples  of  hundreds  of  profession¬ 
al  engineers,  the  difference  in  median  salary  between  engineers  in  Columbus  and  Detroit  (a 
distance  of  less  than  200  miles)  was  more  than  20Z.  A  20Z  median  salary  differential  was 
also  shown  between  hundreds  of  professional  engineers  in  Houston  and  San  Antonio,  approx¬ 
imately  200  miles  apart.  In  the  suaaaer  of  1980,  the  residents  of  Atlanta  paid  4.25  cents 
per  kilowatt-hour  of  electricity,  whereas  New  Yorkers  paid  11.77  cents  per  kilowatt-hour,  or 
177Z  more.  During  the  same  period,  the  rate  per  kilowatt-hour  in  Indianapolis  was  4.96 
cents  while  175  miles  away  in  Columbus  it  was  7.77  cents  or  57Z  more.  (Based  on  a  survey  by 
the  National  Association  of  Regulatory  Utility  Comaissioners) .  Material  costs  depend  on 
extraction  and  processing  labor  costs,  processing  energy  costs,  and  transportation  costs, 
*p|  all  of  which  are  variable  geographically  and  temporally. 


V,  v  7'  t*  *  J 
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B.  PURPOSE/OBJECTIVE 


II.  CONTENTS  AND  SCOPE 


This  report  presents  quantitative  meth¬ 
odologies  for  identifying  the  cost  variabil¬ 
ity  within  the  individual  system  elements,  a 
methodology  for  aggregating  tea  estimated 
uncertainties  in  the  costs  of  the  system 
elements,  and  a  methodology  for  depicting 
the  system  cost  uncertainty  by  a  range  of 
system  costs  with  their  associated  prob¬ 
abilities  of  occurrence.  Thus,  the  inherent 
uncertainties  prevailing  in  a  system  and  in 
its  costs  would  be  reflected  by  a  cost-prob¬ 
ability  relationship  rather  than  as  an 
ambiguous  point  cost  estimate. 

Two  basic  techniques  are  currently  used 

for  generating  the  desired  cost-probability 

relationships:  the  Monte  Carlo  approach  and 

12  3 

the  Method  of  Moments  approach.  *  ’  Both 
of  these  methods,  however,  are  of  such  com¬ 
plexity  that  the  use  of  a  computer  (or  at 
least  a  programmable  calculator)  is  essen¬ 
tial.  Approximately  500  computer  iterations 
are  required  by  the  Monte  Carlo  method  in 
order  to  generate  a  relatively  smooth  cost- 
probability  relationship.  The  need  for  a 

simple,  straightforward  technique  for 

generating  a  quantitative  estimate  of  the 
cost  uncertainty  (risk)  has  long  been 

recognized.  The  objective  of  this  report  is 
to  describe,  in  detail,  the  direct,  quanti¬ 
tative  methodologies  developed  for  deriving 
subsystem  cost  uncertainties,  aggregating 
these  uncertainties  into  a  total  system  cost 
uncertainty,  and  finally  deriving  a  prob¬ 

ability-related  cost  estimate. 


The  subsequent  methodology  for  determin¬ 
ing  total  system  co3t  risk  consists  of  three 
distinct  sequential  steps  or  areas.  The 
first  area  focuses  on  methods  for  estimating 
.the  uncertainty  associated  with  the  cost  of 
individual  system  elements.  Methods  are 
developed  and  illustrated  for  determining 
these  cost  uncertainties  when  the  cost  esti¬ 
mates  are  derived  by  any  of  the  four  basic 
costing  approaches:  parametric,  engineering 
(bottom-up/grass  roots),  analogy,  or  con¬ 
stant  multiplier  (factor).  The  major 
sources  of  uncertainty  that  are  ultimately 
manifested  in  cost  uncertainty  are  treated 
within  each  costing  approach.  These  sources 
of  cost  uncertainty  are  identified  in  Table 
1  for  each  of  the  four  costing  approaches. 
In  the  parametric  costing  approach,  the  two 
major  sources  of  uncertainty  considered  are 
the  uncertainty  in  the  independent  parameter 
(e.g.,  subsystem  weight*)  and  the  uncer¬ 
tainty  in  the  cost  estimating  relationship 
(CER).  In  the  engineering  approach,  the 
uncertainty  in  the  man-hours  (or  quantity  of 
material  required)  is  considered  as  well  as 
the  uncertainty  in  the  corresponding  wage 
rate  (or  cost  per  unit  quantity  of 
material).  In  the  analogous  or  scaling 
approach,  the  uncertainty  in  the  system 
parameter  (e.g.,  weight,  power)  and  the 
uncertainty  in  the  scaling  exponent  are 
considered.  In  the  constant  multiplier  or 
factor  approach,  the  only  source  of  uncer¬ 
tainty  is  in  the  estimate  of  the  system 
parameter  which  is  multiplied  by  a  fixed 
factor  to  obtain  the  estimated  cost. 


*  Weight  really  is  a  dependent  vsriable,  but  because  cost  is  often  highly  correlated  with  it, 
it  ia  frequently  used  in  CERs  as  an  expedient  surrogate  for  a  normalized  mix  of  independent 
•ystem  parameter*,  which  are  also  highly  correlated  with  it. 
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Table  1.  Major  Sources  of  Cost  Uncertainty  for  Four  Basic  Costing  Approaches 


System  Element 

Costing  Approach  Equation _  Sources  of  Uncertainty 


1.  Parametric 


Cost  »  a  +  bPc 


System  parameter  P 

Cost  model  (std.  error  of  eat.) 


2.  Engineering  Cost  =  (MH)  x  (Rate) 

Cost  =  (Mat)  x  (Price) 


3. 


Analogy 


Cost^  /  Pi  \ 
Cost^  \  P2  / 


or 

Cost^ 


4.  ’’actor  Cost  =  P  x  K 


Man-hours  required 
Rate  (skill  level) 

Material  required  (quantity) 
Price  (per  unit  quantity) 


System  parameter  P^ 
Scaling  exponent  Y 


System  parameter  P 


Factors  contributing  to  cost  uncertain¬ 
ty,  such  as  technological  developments 
required,  funding  stretch-out  or  schedule 
slippage,  and  design  changes  are  not  specif¬ 
ically  considered.  However,  they  are 
intriosically  incorporated  in  a  normalized 
manner  within  the  historic  data  poincs  on 
which  the  parametric  CSRs  are  based  and  also 
in  a  similar  manner,  in  the  known  system 
cost  (C2),  and  within  the  exponent  in  the 
analogous  system  costing  approach. 

4 

*  The  second  area  in  the  overall  methodol¬ 

ogy  development  presents  a  procedure  for 
aggregating  the  individual  system  element 
cost  uncertainties  that  were  derived  by  one 
of  the  four  approaches.  The  result  of  the 
aggregation  is  a  composite  uncertainty  that 
reflects  the  uncertainty  (risk)  associated 
with  the  total  system  cost.  Application  of 
the  aggregation  methodology  developed  is 
illustrated  by  an  example. 


The  third  area  presents  a  methodology 
for  developing  a  cost  versus  probability 
relationship.  It  also  includes  some  guide¬ 
lines  for  deriving  a  meaningful,  specific, 
R-ACE  from  the  cost-probability  relationship 
for  program  funding  purposes.  The  methodol¬ 
ogies  developed  are  then  summarized,  and 
future  efforts  for  enhancing  cost  estimating 
methodologies  are  identified. 

III.  METHODOLOGIES  FOR  ESTIMATING  COST 
UNCERTAINTY  WITHIN  SYSTEM  ELEMENTS 

The  methodologies  for  estimating  the 
cost  uncertainty  within  the  individual  sys¬ 
tem  elements  under  the  four  approaches  shown 
in  Table  1  depend  on  estimates  of  the  mean  x 
and  standard  deviation  a  of  the  cost¬ 
driving  parameters.  The  following  section 
describes  two  techniques  for  generating 
values  for  the  mean  and  standard  deviations 
from  engineering  estimates  of  the  cost¬ 
driving  parameters.  The  two  are  the  beta 
distribution  aod  the  triangular  distribution 
techniques. 
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A.  TECHNIQUES  FOR  GENERATING  QUANTITATIVE 
ESTIMATES  OF  COST-DRIVING  VARIABLES  AND 
THEIR  UNCERTAINTIES 

1.  BETA  DISTRIBUTION  TECHNIQUE 

This  technique  traces  its  origin  to  PERT 
(Program  Evaluation  and  Review  Technique) 
where  it  was  widely  used  to  generate  time 
estimates  for  events  in  a  scheduling  net¬ 
work.  In  the  beta  distribution  technique  as 
used  here,  three  estimates  of  a  system  cost- 
-driving  parameter  are  solicited  from  the 
knowledgeable  engineering  specialist  most 
knowledgeable  about  that  system:  low,  most 
likely,  and  high.  The  low,  or  optimistic, 
estimate  should  correspond  approximately  to 
a  value  that  would  be  realized  only  under 
the  most  fortuitous  circumstances  -  a  sub¬ 
jective  probability  somewhere  in  the  0.01  to 
0.10  range.  The  most  likely  estimate  is 
just  that  -  the  mode.  The  high,  or  pess¬ 
imistic,  estimate  should  correspond  to  a 
value  that  reflects  the  ultimate  working  of 
Murphy's  Law  -  a  subjective  probability  in 
the  0.99  to  0.90  range.  Thus,  if  the  three 
values  are: 

a  “  low  estimate 
m  “  most  likely  estimate 
b  “  high  estimate 


4 

distribution  technique.  In  the  tri¬ 
angular  technique,  the  identical  estimates  - 
low  (a),  most  likely  (m),  and  high  (b)  - 

estimates  of  the  cost  driving  parameter 
(e.g.,  weight)  are  obtained  from  the 
knowledgeable  system/ subsystem  specialist. 
The  mean  x  can  then  be  estimated  by 

x  =  j  (a  +  m  +  b)  (3) 

and  the  standard  deviation  O  by 

O  =  / yj  [ (b  -  a)^  +  (m  -  a)(m  -  b)]  (4) 

Generally,  subsystem  characteristics, 
performance  requirements  or  design  con¬ 
straints  enable  the  knowledgeable  subsystem 
specialist  to  readily  arrive  at  the  low  and 
high  estimates.  The  most  likely  estimate 
usually  presents  the  greatest  difficulty  be¬ 
cause  of  a  significant  degree  of  uncertainty 
(indif ferance)  in  the  broad  mid-range  area. 
The  nature  of  the  computations  for  x  and  O, 
however,  tends  to  be  forgiving  and  reduces 
the  percentage  of  error  in  estimating  m. 
Figure  1  shows  the  general  uncertainty  or 
indifference  in  providing  an  estimate  of  the 
most  likely  value,  m. 


then  the  mean  value  x  can  be  estimated  by 

x  ■  7  (a  +  4m  +  b)  (1) 

b 

and  the  standard  deviation  about  the  mean 
can  be  estimated  by: 

°x  -  £  <b  -  a)  (2) 

2.  TRIANGULAR  DISTRIBUTION  TECHNIQUE 

The  triangular  distribution  technique  is 
generally  preferred  over  the  more  common  beta 


A  measure  of  the  sensitivity  of  x  and  O 
to  the  selection  of  m  may  be  seen  from  sn 
analysis  of  the  three  triangular  distribu¬ 
tions  shown  in  Figure  2.  It  assumes  that 

curve  A  is  selected  when  “he  actual  correct 
values  are  depicted  by  curves  B  or  C. 
Table  2  shows  the  sensitivity  of  the  cal¬ 
culated  values  of  x  and  O  with  respect 
tothe  assumption  of  m. 
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Table  2.  Sensitivity  of  Calculated  x  and  o 
with  Respect  to  Assumption  of  m 


Curve 

a 

m 

Am 

b 

A 

50 

95 

0 

140 

S 

50 

72.5 

+  31.0% 

140 

C 

50 

117.5 

-•  19.1% 

140 

The  refinement  of  intentionally  incor¬ 
porating  substantial  skewness  into  the  dis¬ 
tribution  by  the  estimate  of  the  most  likely 
value  does  not  appear  to  be  warranted. 
However,  if  there  are  recognizable  and 
justifiable  reasons  for  estimating  the  most 
likely  value  at  other  than  approximately  the 
mid-range  point,  this  certainly  should  be 
done. 

B.  PARAMETRIC  COSTING  APPROACH 

This  section  describes  and  illustrates 
an  algebraic  technique  for  estimating  the 
uncertainty  within  a  parametrically  derived 
cost  estimate. 

1.  PROBLEM 

Parametrically  derived  cost  estimates 
generally  have  two  major  sources  of  uncer¬ 
tainty:  the  uncertainty  inherent  in  the 
parametric  descriptor  of  the  system  (e.g., 
weight),  and  the  uncertainty  associated  with 
the  cost-estimating  regression  equation, 
which  uses  the  parametric  descriptor  to 
derive  the:  cost  estimate.  The  interrela¬ 
tionship  between  these  two  basic  sources  of 
uncertainty  is  shown  in  Figure  3. 

The  essence  of  the  problem  is  how  to 
propagate  the  uncertainty  in  the  independent 
parameter  that  is  used  to  predict  the  cost, 
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through  the  uncertainty  in  the  cost  esti¬ 
mating  equation,  so  as  to  obtain  an  estimate 
of  the  resultant  composite  uncertainty  in 
the  derived  cost  estimate. 

2.  ASSUMPTIONS / REQUIREMENTS 

The  following  assumptions  are  implicit 

in  the  proposed  methodology: 

a.  An  estimate  of  the  uncertainty  in 
the  independent  variable  (e.g., 
weight)  can  be  generated  (usually  by 
beta  or  triangular  distribution 
techniques  such  as  described  in  the 
preceding  section). 

b.  A  measure  of  the  uncertainty  in  the 
cost-predictive  regression  equation 

is  obtainable  in  terms  of  the  stan- 
5  6  7  8 

dard  error  of  estimate.  ’  ’  ’ 

c.  The  uncertainty  distributions 
associated  with  both  the  indepen¬ 
dent  variable  and  the  regression 
equation  are  symmetrical.  (Neither 
the  Monte  Carlo  approach  nor  the 
Method  of  Moments  approach  is 
restricted  to  this  assumption.) 

d.  The  independent  variable  is  within 
the  predictive  range  (bounds)  of  the 
regression  equation.  Extrapolation 
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beyond  the  bounds  of  the  regression 
equation  could  introduce  significant 
additional  sources  of  uncertainty 
that  are  not  treated  here. 

e.  The  cost-predictive  equation  is  of 
the  general  form: 

Y  •  a  +  bxc 

For  example, 

Y  -  a  +  bxc 

Y  ”  bxC  (a,  the  Y  intercept 

is  equal  to  zero) 

Y  “  a  +  bx  (c  is  equal  to  1) 

Y  ”  bx  (a  is  equal  to  zero, 

and  c  is  equal  to  1)' 


3.  METHODOLOGY  FOR  DETERMINING  THE  UNCER¬ 
TAINTY  IN  A  PARAMETRICALLY  DERIVED  COST 

ESTIMATE 

The  application  of  the  algebraic 
relationship  requires  the  following  data: 

a.  For  the  independent  variable  (e.g., 

system  weight),  an  estimate  of  the 

mean  value  x,  and  an  estimate  of  the 

standard  deviation  a  . 

x 

b.  A  cost  predicting  regression  equation 
(of  the  general  form  Y  “  a  +  bxC) 
together  with  an  estimate  of  the 
standard  error  (SE)  about  the  re¬ 
gression  curve. 


141 


Knowing  both  the  mean  and  standard 

deviation  of  the  independent  variable  (e.g., 
weight),  as  well-  as  the  regression  line 
equation  with  its  standard  error  of  esti¬ 
mate.  Then  the  standard  deviation  of  the 
dependent  variable  (e.g.,  cost)  can 

be  estimated  by  the  relationship:* 

2,  2 

02  ■  SE2-+  7  ["(x  +  0  )c  -  (x  -  0  )cl  (5) 
Y  4  L  x  x  J 

It  should  be  noted  that  this  relation¬ 
ship  is  independent  of  a  (the  Y  intercept) 
and  so  will  hold  true  for  regression  equa- 

Q 

tions  of  the  form  Y  =  bx  .  In  the  linear 

case  Y  ■  a  +  bx,  it  reduces  to  a  standard 

result.  The  application  of  this  methodology 
is  illustrated  by  the  following  example. 

4.  APPLICATION  OF  PARAMETRIC  COST  UNCER¬ 
TAINTY  DETERMINATION 

The  following  example  is  based  on  the 
Telemetry,  Tracking  and  Command  (TT&C)  sub¬ 
system  data  for  generating  first  unit  cost 
and  was  extracted  from  the  Unmanned  Space¬ 
craft  Cost  Model,  SD-TR-81-45,  dated  June 
1981  which  was  prepared  by  Air  Force  Space 
Division/ ACC.  The  TTiC  subsystem  design 
weight  of  81.6  lb  found  on  page  VII-3  of  the 
subject  document  was  modified  by  the  data 
for  triangular  distribution  parameters  shown 
in  Table  D-l,  page  D-4,  as  follows  so  as  to 
represent  typical  data  that  might  be 
encountered  in  an  actual  case: 

Basic  design  weight:  81.6  lb 
Low  estimate  (-  50Z)  40.8  l'o  *  a 

Mode  or  most  likely  estimate 
(+  2.9Z)  84.0  lb  -  ra 

High  estimate  (  +  58Z)  128.9  lb  »  b 


The  mean  value,  assuming  a  triangular 
distribution,  can  be  derived  by  Eq.  (3),  so 
that  the  mean  weight  in  this  case  is 

x  -  1/3  (40.8  ♦  84.0  +  128.9)  -  84.6  lb 

Likewise,  the  standard  deviation  for  a 
triangular  distribution  can  be  estimated  by 
Eq.  (4),  which  in  this  case  is 

Ox  -  [l/18[(128.9  -  40.8)2+ 

(84  -  40.8)(84  -  128.9) l]1^2  -  18.0  lb. 

The  first  unit  cost-predicting  regress¬ 
ion  equation  for  TT&C  is  given  on  page  IV-11 
of  the  referenced  document  to  be 

Y  -  42.43  v  3p.93x*93 

where  Y  “  First  unit  cost  in 

thousands  of  FY  79  dollars 
and 

x  ”  TT&C  weight  in  pounds 

By  substituting  the  mean  weight  of 
84.6  lb  for  x  in  this  equation  and  solving, 
an  estimated  first  unit  cost  of  Y  *  42.43  + 
35. 93(84. 6)'93  -  |>2270.4  is  obtained.  The 
standard  error  of  estimate  (SE)  for  this 
equation  is  reported  as  713.9.  By  inserting 
the  corresponding  values  into  the  cost 
uncertainty  estimating  equation  (5),  the 
standard  deviation  for  the  TT&C  first  unit 
cost  is.  found  to  be 

0Y2  -  713.92  +  [(84.6  +  18.0)"93  - 

.93]  2 

(84.6  -  18.0) 


*  R.  H.  Huddlestone,  Prediction  Error  Statistics  for  a  Nonlinear  Cost  Risk  Model,  The 
Aerospace  Corporation"!  Interoffice  Correspondence  ( 13  January  1982) .  (Not  available  for 
public  distribution).' 
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or 


2.  ASSUMPTIONS/REQUIREMENTS 


Gy  =■  $839.2 

The  contribution  of  the  uncertainty  in 
the  weight  (a^  “  18.0  lb)  resulted  in  an 
increase  in  uncertainty  of 


over  that  attributable  to  the  standard  error 
of  estimate  alone. 


C.  ENGINEERING  (BOTTOM-UP/GRASS  ROOTS) 
COSTING  APPROACH 

The  engineering  approach  is  probably  the 
most  widely  used  technique  for  preparing 
cost  estimates  of  systems  chat  are  in  the 
latter  stages  of  development. 

1.  PROBLEM 


The  following  are  required  for  the 
implementation  of  this  uncertainty  prop¬ 
agating  methodology: 

a.  An  estimate  of  the  quantity  required 
a  along  with  a  measure  of  the  uncer¬ 
tainty  in  this  estimate  in  terms  of 

the  standard  deviation  G  .  The 

a 

estimated  standard  deviation  can  be 
derived  by  either  beta  or  triangular 
distribution  techniques  as  previous¬ 
ly  described. 

b.  An  estimate  of  the  rate  per  unit 

quantity  b  together  with  an  estimate 

of  its  standard  deviation  a,  . 

b 

This  too  can  be  derived  by  beta  or 
triangular  distribution  techniques 
as  previously  mentioned. 


Engineering  cost  estimates  focus 
primarily  on  labor  and  material  costs  and 
secondarily  on  energy  and  processing  costs. 
Cost  estimates  of  system  elements  are 
usually  derived  by  multiplying  the  quantity 
estimated  to  be  required  times  a  unit  rate 
(e.g.,  man-hours  times  dollars  per  hour; 
pounds  times  dollars  per  pound,  units  times 
dollars  per  unit).  In  this  case,  the  two 
major  sources  of  uncertainty  are  the  uncer¬ 
tainty  in  the  estimate  of  the  required  quan¬ 
tity  (e.g.,  man-hours,  pounds),  and  the 
uncertainty  in  the  estimate  of  the  cost  per 
unit  (e.g.,  dollars  per  man-hour,  dollars 
per  pound).  The  essence  of  this  problem  is 
how  to  combine  the  uncertainty  in  the  quan¬ 
tity  required  with  the  uncertainty  in  the 
dollar  rate  per  unit  quantity  so  as  to 
obtain  an  estimate  of  the  uncertainty  in  the 
product  cost. 


c.  The  uncertainty  distributions 
associated  with  both  the  quantity 
estimate  and  the  rate  per  unit  quan¬ 
tity  are  symmetrical. 

d.  The  quantity  and  rate  per  unit  quan¬ 
tity  are  assumed  to  be  independent. 

3.  METHODOLOGY  FOR  DETERMINING  THE  UNCERTAIN- 
UNCERTAINTY  IN  AN  ENGINEERING  (BOTTOM-UP/ 
GRASS  ROOTS)  COST  ESTIMATE 

As  stated  in  the  assumptions,  in  order 

to  apply  this  methodology,  it  is  necessary 

to  have  estimates  of  the  quantity  (mean)  a 

and  its  standard  deviation  G  ,  as  well 

a 

as  the  rate  per  unit  quantity  (mean)  b  and 
its  standard  deviation  G,  .  These  values 

D 

are  used  to  calculate  the  coefficient  of 
variation  (called  the  "fractional  standard 
deviation"  by  some  authors)  for  both  the 
quantity  and  the  rate.  The  coefficient  of 


variation  S  is  the  standard  deviation 
divided  by  the  mean.  Thus, 


Rate : 


0.167 


Coefficient  of  variation  for  quantity: 

a 

S  -  —  (6) 

a  a 


Coefficient  of  variation  for  rate: 


The  cost  is,  then,  the  product  (P6)  that 
results  frcm  multiplying  the  quantity  a 
times  the  rate  per  unit  quantity  b 

Pt  =*  a  x  b 

The  standard  deviation  Op  about  the 

product  P$  is  then  found  by: 

op  -  ab  / S2  +  S2  (7) 

4.  APPLICATION  OF  COST  UNCERTAINTY  DETERM¬ 
INATION  IN  THE  ENGINEERING  APPROACH 

If  5000  diverse  engineering  man-hours  a 

were  expected  to  be  required  to  develop  a 

specific  element  with  a  standard  deviation 

a  of  500  man-hours,  and  the  mean 

a 

burdened  rate  per  engineering  man-hour  b  was 
estimated  to  be  $60  per  hour  with  a  standard 
deviation  of  fclO  per  hour,  then  Che 

estimated  cost  of  developing  the  element  is 

Pj  *  a  x  b 
Pt  -  5000  x  60 
Pt  -  300,000 

The  coefficients  of  variation  are 

a 

Quantity:  sa  “  ~  *  JjJoo  “  °- 1° 


a 

n  b  10 

o  b  1>0 

and  the  standard  deviation  for  the  product 
of  a  times  b  is 


5000  x  60/  0.102  +  0.1672 


Op  =•  658,400 

Whereas,  the  individual  coefficients  of 
variation  were  0.10  and  0.167,  the 
coefficient  of  variation  in  the  resulting 
product  is 


°P  a  58,400 
P$  “  300,000 


0.195 


or  greater  than  that  of  either  of  the 
factors . 


D.  ANALOGOUS  (SCALING)  COSTING  APPROACH 

The  analogous  or  scaling  costing  approach 
is  used  in  cases  where  sufficient  historical 
data  are  lacking  so  that  meaningful/valid 
CERs  cannot  be  developed  by  statistical 
(regression)  techniques,  but  where  there  is 
a  close  similarity  between  the  system 
(subsystem)  being  costed  and  an  existing 
system  (subsystem)  whose  cost  is  known. 
This  approach  is  widely  used  in  the  chemical 
process  industry  to  generate  preliminary 
cost  estimates  for  new  chemical  plants  or 
equipment  based  on  the  known  cost  of  a  pilot 
plant  or  smaller  scale  installation.  It  is 
commonly  referred  to  in  the  literature  as 
the  "six-tenths  factor"  costing  approach 
even  though  scaling  factors  other  chan  0.6 
are  often  used. 
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1.  PROBLEM 


In  che  analogous  costing  approach,  a 
syacem  similar,  or  as  che  term  implies  - 
analogous  -  to  the  one  being  costed  exists. 
Key  characteristics  (e.g.,  weight,  output), 
proven  by  experience  to  be  highly  correlated 
with  cost,  are  known  for  both  systems  -  the 
analogous  system  and  the  system  being  costed. 
Thus,  the  variables  in  the  cost  estimating 
equation  are 

=»  cost  of  the  new  system  (sought) 

Cq  *  known,  actual  cost  of  old,  analogous, 
existing  system 

P^  *  cost-correlated  characteristic  of  new 
system 

PQ  ■  known,  cost-correlated  characteristic 
of  old,  analogous,  existing  system 
K  ■  experientially  based  scaling  exponent 

and  che  cost  estimating  relationship  is 


known  (no  uncertainty).  The  two  sources  of 
input  uncertainty  in  this  approach  are 
associated  with  the  characteristic  of  the 
new  system  being  costed  P^,  and  with  the 
uncertainty  in  the  scaling  exponent  K. 

A  perusal-  of  exponents  used  to  cost 

chemical  industry  elements  reveals  exponents 

ranging  from  approximately  0.1  to  over  3.0, 

q 

although  most  fall  between  0.4  and  0.9. 
Scaling  exponents  for  fighter  or  transport 
aircraft  subsystems  generally  range  from 
0.70  to  1.0,  with  most  falling  between 
0.8  and  0.9. 10 

2.  ASSUMPTIONS/REQUIREMENTS 

The  following  assumptions  and  require¬ 
ments  underlie  the  application  of  the  cost¬ 
ing  by  analogy  approach: 

a.  An  estimate  of  the  mean  value  of  the 
cost-correlated  characteristic  P^ 
(e.g.,  weight,  power  output)  of  the 
new  system  or  subsystem  is  required 
along  with  an  estimate  of  the  uncer¬ 
tainty  about  the  mean  value  in  terms 
of  the  standard  deviation  Op^. 

b.  An  expected  value  of  the  scaling 
exponent  K  is  required  together  with 
its  uncertainty  in  terms  of  the 
standard  deviation  0R. 

c.  The  uncertainty  distributions 
associated  with  the  characteristic 
P^  and  with  the  exponent  K  are 
both  assumed  to  be  symmetrical. 


3.  METHODOLOGY  FOR  DETERMINING  THE 

UNCERTAINTY  IN  A  COST  ESTIMATE  DERIVED 
BY  THE  ANALOGY  (SCALING)  APPROACH 

This  methodology,  developed  by 
Dr.  R.  H.  Huddlestone*,  differs  from  those 
previously  presented  in  that  the  uncertainty 
in  the  cost  is  not  uniquely  determined  by  a 
specific  equation,  but  rather  by  a  procedure 
as  follows:  The  two  key  sources  of  uncer¬ 
tainty  in  the  analogy  method  are  (1)  the 
estimate  of  the  new  system's  characteristic, 
and  (2)  the  estimate  of  the  scaling  expon¬ 
ent.  Estimates  of  these,  along  with  their 
standard  deviations  as  measures  of  their 
uncertainty,  need  to  be  obtained,  which  can 
be  done  by  the  beca  or  triangular  distribu¬ 
tion  techniques  as  described  in  Sections  III 
A  1  and  III  A  2. 

The  standard  deviations  are  then  applied 

to  the  mean  values  of  P„  and  K  so  as  to 

N 

obtain  +  1  and  -  1  standard  deviation  values 
for  each  of  the  two  parameters  -  four  points 
in  all. 


PN  +  l0PN 

PN  "  laPN 
X  +  la„ 


X  -  la„ 


These  values  are  then  used  to  determine  four 

estimates  of  the  cost  C„,  where 
N 


C 


N 


The  four  values  of  Cjj  are  obtained  by 
substituting  the  following  four  seta  of 
values  into  the  preceding  equation  and 
solving  for  C^: 

1.  PN  "  l0PN>  K  “  loK 

2.  PN  -  lOpN,  X  +  loK 

3.  Pfl  +  lDp^,  X  —  l0£ 

4.  Pjj  +  10pN,  X  +  lOj^ 

The  mean  of  the  four  C^  values  is  then 
obtained,  and  the  standard  deviation  of  the 
four  values  of  C^  from  the  mean  C^  is 
generated.  The  best  estimate  of  the  cost  of 
the  new  system  is  the  mean  value,  and  the 
standard  deviation  thus  calculated  is  the 
sought  estimate  of  the  uncertainty  in  the 
system  cost. 

4.  APPLICATION  OF  COST  UNCERTAINTY 

DETERMINATION  IN  THE  ANALOGY  APPROACH 

A  system  analogous  to  the  one  being 
costed  was  known  to  have  weighed  200  lb 
(Pq)  and  cost  S>10  million  (C^).  The  new 
system  is  expected  to  weigh  800  lb  (P^) 
with  a  standard  deviation  of  200  lb 
(O  ).  The  expected  scaling  exponent 
Xj.  is  estimated  to  be  0.6  with  a  standard 
deviation  of  0.2.  Thus, 

CQ  -  610  M 
PQ  -  200  lb 

P^  (low,  -  lOpN)  -  600  lb 
Pjjg  (expected)  =  800  lb,  CJpN  =200 
P[fH  (high,  +  10pN)  =  1000  lb  and 
XL  (low,  -  laR)  =  0.4 
Kg  (expected)  =0.6,  =  0.2 

X^  (high,  +  laR)  =  0.8 


*  R.  H.  Huddlestone,  Estimated  Error  in  Costing  by  Analogy, 
Interoffice  Correspondence  (7  January  1983).  (Not  available 


The  Aerospace  Corporation, 
for  public  distribution.) 
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Figure  4.  Schematic  of  Bounds  of  System  Cost  C„ 

N 


By  using  the  equation 


C 


N 


K 


and  the  four  preceding  sets  of  values  for 
PN  and  K,  four  estimates  of  are 

obtained.  The  best  estimate  of  CN  can  be 
shown  to  lie  within  the  bounds  of  these  four 
points,  as  illustrated  in  Figure  4.  An 


estimate  of  the  standard  deviation  a... 
can  be  obtained  by  determining  the  standard 
deviation  of  the  value  of  for  these 

four  points  from  the  mean  C^,  as  shown  in 
Table  3. 

Thus,  the  estimated  cost  of  the  new  system 
CN  is  (23.7  million  with  a  standard 
deviation  Oc{J  of  (7.8  million. 


M 


Table  3.  Illustrative  Data  and  Computational  Results  for  Determining 
Estimated  Coat  and  Cost  Uncertainty  by  the  Analogy  Approach 


Point 

PN 

K 

Calc . 

CN 

Dev.  (D)  From 

Mean  CN 

D2 

1. 

Low 

600 

Low 

0.4 

15.52 

8.20 

67.24 

2. 

Low 

600 

High 

0.8 

24.08 

0.36 

0.13 

3. 

High 

1000 

Low 

0.4 

19.04 

4.68 

21.90 

4. 

High 

1000 

High 

0.8 

36.24 

12.52 

156.75 

Total  C^ 

-  94.88 

Total  D2  - 

246.02 

Mean  C^ 

-  23.72 

Then*  CCN 

■ 

1 

7.84 

\ 
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E.  CONSTANT  MULTIPLIER  (FACTOR)  COSTING 
APPROACH 

la  some  instances,  the  cost  o£  subsys¬ 
tems  can  be  estimated  by  multiplying  the 
cost  estimate  of  a  related  system  element  by 
a  fixed  factor(s).  The  factor  might  reflect 
a  spares  cost  factor,  inflation  factor, 
learning  curve  factor,  or  some  ocher  fixed 
parameter.. 

1.  PROBLEM 


multiplied  by  a  constant  is  well 
11  12 

established,  ’  and  is  presented  here 
for  completeness.  The  methodology  consists 
of  multiplying  the  standard  deviation  of  the 
original,  independent  variable  by  the  con¬ 
stant  in  order  to  obtain  the  standard 
deviation  in  the  resulting,  dependent  vari¬ 
able.  Thus,  if  the  initial  cost  is  C,  its 
standard  deviation  O,  and  the  constant 
multiplier  K,  then  the  new  cost  C^  and  its 
standard  deviation  are  found  by: 


In  this  approach,  the  factor  is  fixed  - 
by  edict  or  mathematical  principles.  The 
cost,  which  is  multiplied  by  the  factor, 
however,  does  possess  uncertainty  -  either 
actual  historical  variability  or  developed 
estimated  cost  uncertainty.  The  proolem  in 
this  approach  is  to  ascertain  the  variability 
in  a  cost  developed  by  multiplying  a  cost 
estimate,  that  has  some  level  of  uncertainty 
by  a  fixed  constant. 

2.  ASSUMPTIONS/ REQUIREMENTS 

The  following  are  required  for  the 
implementation  of  the  constant  multiplier 
(factor)  costing  approach: 

a.  An  estimate  of  the  historical  or 
previously  generated  (expected)  cost 
of  the  subsystem/element,  along  with 
a  measure  of  the  uncertainty  about 
the  expected  cost  as  expressed  by 
the  standard  deviation. 


=*  KCl 

(9) 

p— ^ 

O 

11 

(10) 

4.  APPLICATION  OF  COST  UNCERTAINTY 
DETERMINATION  IN  THE  CONSTANT 
MULTIPLIER  APPROACH 

If  a  system  was  estimated  to  cost  $850 
million  in  some  past-year's  dollars  with  a 
standard  deviation  of  $>200  million,  and  the 
inflation  factor  to  bring  the  past  costs  to 
current-year  dollars  is  1.3,  then  the  new 
(current-year)  cost  and  its  standard 
deviation  are  obtained  as  follows: 

K.  -  constant  (e.g.  ,  inflation  factor) 

C^  "  cost  in  past-year  dollars 

C^  *  cost  in  inflated  current-year  dollars 

0^  ■  uncertainty  associated  with  the 

cost  expressed  in  past-year  dollars 
*  uncertainty  associated  with 
adjusted-for-inf lation  cost 


b.  The  uncertainty  distribution  about 
the  expected  cost  is  symmetrical. 

3.  METHODOLOGY  FOR  DETERMINING  THE 
UNCERTAINTY  IN  A  COST  ESTIMATE 
GENERATED  BY  A  CONSTANT  MULTIPLIER 

The  methodology  for  propigsting  the 
uncertainty  in  a  .variable  when  it  is 


and 


C2  ’  KC1 

C^  “  1.3  x  $850  million 
C^  “  $1105  million 

°2  ’  K°1 

“  1.3  x  $200  million 
0,  -  $260  million 
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Standard  Deviation  Vananc e 


Thus,  the  cost  in  current-year  dollars  is  Variable 

found  to  be  &  1 105  million,  with  a  standard 
deviation  of  £260  million. 


IV.  METHODOLOGY  FOR  AGGREGATING  THE  COST 
ELEMENT  UNCERTAINTIES 

In  the  preceding  section,  methodologies 
were  presented  for  developing  cost  uncer¬ 
tainties  when  individual  system  elements  are 
costed  by  one  of  the  four  basic  costing 
approaches.  The  costs  of  Che  individual 
system  elements  are  obviously  aggregated  by 
adding,  but  it  is  not  clear  how  the 
uncertainties  (e.g.,  os)  associated  with 
the  system  element  costs  should  be  combined 
to  reflect  the  uncertainty  in  the  aggregated 
cost.  This  section  identifies  a  methodology, 
along  with  its  rationale,  for  aggregating 
the  cost  uncertainties. 


b 


c 


_d_ 
Sum  S 


c 


O 


2 

d 


The  variance  about  the  sum  a*  then,  is 


2  2  2  2  2 

0C  =*  o  +  a.  +  a  +o, 
S  a  b  c  d 


The  standard  deviation  about  the  sum 
°s  ma^  cben  be  derived  by  taking  the 
square  root  of  both  sides  of  the  equation; 
thus , 


s  A.  FALLACY  IN  APPLICATION  OF 

CLASSICAL/ CONVENTIONAL  STATISTICS 

Conventional  statistics  holds  chat  if 
variables  are  additive,  then  their  variances 
(the  square  of  the  standard  deviation  or 
2 

a  )  are  also  additive.  Thus,  Che 
variance  about  Che  sum  of  a  series  of  vari¬ 
ables,  each  with  its  own  variance,  is  found 
by  adding  the  variances  as  follows: 

1 


o 


S 


+  o. 


2 

+  O 


c 


+ 


o 


2 

d 


This  is  the  conventional  root-sunr-square 
(RSS)  relationship.*  What  happens,  however, 
when  this  relationship  is  applied  to  system 
cost  data? 


Suppose  a  system  is  composed  of  three 
independent  elements,  each  with  a  cost  of  50 
and  a  standard  deviation  about  the  cost 


*  A  refinement  of  the  RSS  technique  focuses  on  the  independence  or  nonindependence  of  the 
variables  being  aggregated.  If  they  are  indeed  all  independent,  then  the  conventional, 
straightforward  RSS  equation  applies.  However,  if  they  are  nonindependent  (correlated), 
then  an  additional  complexity  arises  -  covariance  term(s)  need  to  be  added  to  the  sums  of 
the  variances.** 

**  R.  H.  Huddlestone,  Estimated  Error  in  Total  System  Cost,  The  Aerospace  Corporation, 
Interoffice  Correspondence  (30  November  1982)  discusses  this  issue.  (Not  available  for 
public  distribution.) 
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equal  co  102  of  the  cost,  or  5.  The  total 
syscem  cost  and  its  standard  deviation  would 
be  derived  chusly: 


Flemenc 

Cosc 

Scandard  Deviacion 

’ 

50 

51 

2 

50 

52 

•  3 

50 

53 

Tocal  cosc 

»  156 

The  standard  deviation  about  the  total 
cost  computed  by  applying  the  RSS 
approach  would  be 


The  standard  deviation,  which  initially 
amounted  to  102  of  the  cost  of  each  element 
is,  by  the  application  of  the  RSS  technique, 
reduced  co  5. 82  of  the  sum. 

Now,  if  the  system  were  twice  as  large 
and  consisted  of  six  racher  chan  chree  like 
elemencs,  each  of  which  again  had  a  scandard 
deviacion  equal  co  102  of  Che  mean,  Chus, 


Element 

Cosc 

Scandard  Deviacion 

1 

50 

51 

2 

50 

52 

3 

50 

53 

4 

50 

54 

5 

50 

55 

6 

50 

56 

Tocal  Cosc 

-  300 

Chen,  Che  scandard  deviacion  abouc  Che  cocal 
cosc  using  Che  RSS  approach  would  be 


or  12.25.  This  is  buc  4.12  of  Che  sum. 


If  che  syscem  were  cruly  gigancic  and 
complex,  such  chac  ic  consisced  of  24 
scaciscically  independenc  elemencs  under  che 
above  assumpeions,  Chen  che  cocal  cosc  would 
be  1200,  and  che  scandard  deviacion  using 
che  rooc-sum-square  approach  would  be  24.49, 
or  now  only  2.02  of  che  cocal  cosc!  These 
examples,  which  show  che  effecc  of  applying 
convencional  RSS  cechnique3,  are  summarized 
in  Table  4  below. 

Table  4.  Potential  Irapacc  of  Using  RSS 
Technique  for  Cosc  Uncercaincy 
Aggregation 


No.  of 

Tocal 

Uncercaincy 

Uncertainty  O 

Elemencs  Cosc 

a 

2  of  Tocal  Cosc 

3 

150 

8.66 

5.8 

6 

300 

12.25 

4.1 

24 

1200 

24.49 

2.0 

These  daca  show  chac  as  che  syscem  becomes 
larger,  (e.a.,  mere  expensive  and  complex) 
che  percent  uncercaincy  cends  to  decrease! 
This  is  contrary  to  all  costing  experience. 
Something  is  wrong.  An  examination  of  what 
is  actually  occurring  will  reveal  the  source 
of  this  fallacy. 

B.  SOURCE  OF  INAPPLICABILITY  OF 
CONVENTIONAL  STATISTICS 

1.  NATURE  OF  COMPETITIVE  PROCUREMENT  PROCESS 

When  proposals  for  the  development 
and/or  production  of  a  syscem  are  received 
from  several  qualified  contractors,  cosc  is 
invariably  a  criterion  for  selecting  Che 
winner.  Not  necessarily  Che  lowest  cost  - 
buc  racher  a  reasonable,  credible  cost. 
This  obviously  puts  pressure  on  the 
competitive  bidders  Co  develop  low  but 
realistic  bids.  However,  in  most  system 
procurements,  there  is  a  significant  element 


of  uncertainty*  This  is  reflected  by  the 
inclusion  of  prototype  development  and  tests 
into  the  program.  The  tests  are  •  in  essence 
demonstrations  to  ascertain  whether  the 
manifold  uncertainties  pervading  the  system 
have  been  successfully  resolved.  Tests  are 
expensive.  How  many  tests  will  be  required 
reflects  just  one.  aspect  of  contractor 
optimism.  Some  level  of  optimism 
(confidence  in  the  engineering  staff's  tech¬ 
nical  competence)  must  be  displayed.  A  bid 
based  on  pessimistic  outcomes  would  surely 
lose  to  a  more  success-oriented  competitor. 
On  the  other  hand,  the  compulsion  to  submit 
a  low  bid  based  on  substantial  optimism 
would  be  prudently  balanced  by  experiential- 
ly  derived'  knowledge  of  the  existence  of  a 
potentially  malevolent  reality  (Murphy's 
Law!).  Consequently,  modest  reserves  for  a 
limited  number  of  adverse  events  are  usually 
incorporated  into  the  bid.  Thus,  the  nature 
of  the  competitive  procurement  process 
results  in  bids  that  (1)  tend  to  be  somewhat 

low,  based  on  both  an  optimistic  view  of  the 

\ 

contractor's  technical  competence  and  pre¬ 
vailing  competitive  forces,  and  (2)  belong 
to  a  two-tailed  distribution  inasmuch  as  the 
cost  could  be  lower  if  the  program  were  so 
successful  that  the  reserves  were  not 
needed;  or  higher,  if  the  amount  of  reserves 
were  inadequate.  Invariably,  the  uncertain¬ 
ties  surrounding  the  costs  of  all  of  the 
individual  elements  that  are  aggregated  to 
produce  the  final  bid  are  two-tailed,  even 
though  some  tails  may  be  highly  skewed. 

Thus,  in  aggregating  the  costs  and  their 
uncertainties,  a  root-sum-square  approach 
would  appear  valid.  The  high-tailed  cost 


uncertainties  would  be  offset  by  the 
low-tailed  cost  uncertainties.  The  popula¬ 
tion  would  tend  towards  normality,  and  the 
Central  Limit  Theorem  would  hold. 

2.  MANAGEMENT  PRINCIPLES/DECISION  DRIVERS 

Once  a  contract  is  awarded,  what  was  up 
to  that  point  a  "normal"  distribution 
becomes  a  truncated  (one-sided)  distribu¬ 
tion,  and  the  root-sum-square  approach  to 
aggregating  the  uncertainties  is  no  longer 
valid.  That  portion  of  the  cost  distribu¬ 
tion  which  lies  below  the  mean  or  expected 
cost  disappears,  so  there  are  no  offsetting 
values  for  the  high  portion  of  the  distribu¬ 
tion.  The  reason  for  this  is  based  on 
program  management  motivators  or  decision 
drivers.  The  program  manager  is  acutely 
aware  that  the  systems  being  procured  have  a 
higher  intrinsic  value  than  what  is 
reflected  in  their  cost  (or  else  their 
acquisition  would  be  uneconomic  and 
imprudent).  Both  his  and  his  company's 
reputation  depend  on  the  reliable  perfor¬ 
mance  of  the  developed  system  over  its 
expected  life  -  but  in  the  case  of  space 
systems  -  with  stringent  weight  and/or 
volume  constraints.  The  anticipated  re¬ 
liability  is  substantiated  by  failure  mode 
analyses  which  pinpoint  weak  links. 
Engineers  responsible  for  elements  of  the 
space  system  invariably  identify  changes 
that  would  enhance  reliability  and  improve 
the  system,  but  usually  at  an  increase  in 
cost,  weight,  and/or  volume.  Thus,  if  the 
cost  is  underrunning,  a  prudent  manager 
would  opt  to  use  the  available  funds  to 
eliminate  the  ever-present  weakest  link  and 
so  enhance  the  system's  reliability,  life, 


or  ocher  key  attributes.*  It  is  virtually 
inconceivable  that  a  program  manager  would 
decide  to  come  ir.  below  the  contract  cost 
while  weak  links  exist  that  could  jeopardize 
the  total  success  of  the  system.  (How  much 
testing  and  reliability  ia  enough?) 

Furthermore,  the  level  of  contractor 
optimism  evoxed  by  a  competitive  procurement 
process  operates  to  reduce  the  probability 
of  such  a  highly  successful  program  chat  a 
cost  underrun  would  result.  (Some  sole 
source  procurements  without  cost-depressing 
competitive  pressures  have  resulted  in 
underruns,  but  this  is  deemed  to  be  due  to 
the  submittal  of  a  more  realistically  high 
bid  than  would  normally  be  the  case  in  a 
competitive  procurement.)  The  pressures  on 
the  program  manager  personally  and  on  Che 
contractor  in  general  to  deliver  the  "best" 
possible  system,  coupled  with  the  ubiquitous 
weak  links,  preclude  cost  underruns. 
Consequently,  the  system  costs  incurred  are 
noc  governed  by,  nor  display,  randomness 
about  the  contract  awarded  amount.  The  RSS 
technique  in  which  variances  are  summed, 
assumes  a  randomizing  nature  at  work,  or  an 
"averaging"  of  the  high  side  values  with 
Chose  below  the  expected  (i.e.,  mean) 
value.  This  does  noc  occur  here.  There  are 
no  "low"  values.  If  the  costs  of  the 
elements  were  normally  distributed  so  Chat 
some  were  higher  chan  expected  while  others 
were  lower  (thus  producing  a  bell  shaped 
histogram  or  curve),  then  the  conventional 


RSS  approach  might  apply,  but  this  is  not 
the  case.  The  awarded  amount  -  usually  the 
bid  price  (slightly  modified  during  contract 
negotiations)  -  which  was  two-tailed  in  it' 
development,  becomes  a  floor  once  the  award 
is  made.  The  two-tailed,  chance  element 
chat  underlies  the  RSS  technique  no  longer 
prevails.  Root-sum-squaring  should  not  be 
used.  It  generates  wrong  estimates  of  the 
standard  deviation  because  "averaging"  does 
not  occur. 

The  foregoing  assessment  of  the  program 
manager  applies  equally  down  the  organiza¬ 
tional  line  to  the  subsystem  project 
engineers  and  subcontractors.  Once  a  cost 
goal  has  been  sec  or  subcontract  award  made 
(based  again  on  a  measure  of  optimism),  that 
then  becomes  the  cost  floor.  The  best  sub¬ 
system  possible  will  be  developed  for  the 
designated  cost.  However,  any  adversity 
encountered,  beyond  that  anticipated  in 
preparation  of  the  proposed  bid,  will  likely 
result  in  cost  overruns.  Thus,  the 
bid/award  cost  will  not  be  underrun.  It  can 
only  be  mec  or  overrun.  .  This  negates  the 
applicability  of  the  root-sum-square  tech¬ 
nique  for  aggregating  system  cose 
uncertainties. 

C.  RECOMMENDED  APPROACH 

The  independent  parameters  (e.g.,  sub¬ 
system  weight,  CER  standard  error  of  esti¬ 
mate)  on  which  cost  estimates  are  based 


*  Major  General  Jasper  Welch,  USAF,  Assistant  Deputy  Chief  of  Staff  for  Research,  Development 
and  Acquisition,  writing  in  the  December  1982  issue  of  Electronic  Business  (pages  55,  56) 
admonished  contractors  for  merely  meeting  rather  than  substantially  exceeding  negotiated 
MTBF  specifications. 
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possess  an  inherent  two-tailed  probability 
distribution  (±  0).  The  beta  and 

triangular  distribution  techniques'  described 
provide  methods  for  estimating  a  two-tailed 
uncertainty  about  a  mean  or  expected  value. 
The  methodologies  described  for  propagating 
the  uncertainties  in  the  independent  vari- 
ables  within  the  four  costing  approaches 
result  .in  the  development _  of  two-tailed 
uncertainty  distributions  about  each  element 
or  subsystem  cost  derived.  Now,  the  purpose 
of  aggregating  the  element/subsystem  costs 
and  their  uncertainties  is  to  arrive  at  a 
total,  composite  cost  along  with  its  uncer¬ 
tainty  that  will  reflect  actuality  -  what 
really  may  occur.  Conventionally,  aggrega¬ 
tion  of  two-tailed  uncertainties  is  validly 
done  by  means  of  the  RSS  technique  -  RSS 
alone  if  the  variables  whose  uncertainties 

>  -  are  being  aggregated  are  independent,  RSS 

plus  a  covariance  factor  if  the  variables 
are  nonindependent.  But,  as  described  in 
the  preceding  section,  forces  prevail  that 
eliminate  the  lower  tail,  thus  negating  the 
applicability  of  a  RSS  approach.  The 
one-sidedness  of  the  cost  incurrence  is  the 
basis  for  the  recommended  approach. 
Specifically,  in  aggregating  the  uncertain¬ 
ties  associated  with  one-sided  cost 
elements,  the  uncertainties  as  reflected  in 

3?  ... 

the  standard  deviation  associated  with  each 
of  the  aggregated  cost  elements  should  be 
added  to  arrive  at  a  standard  deviation  for 
the  aggregated  cost.* 

D.  APPLICATION  OF  AGGREGATION  METHODOLOGY 

The  following  example  is  based  on  an 
actual  space  system  concept.  High,  low,  and 
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most  likely  weight  estimates  were  obtained 
for  each  subsystem  element  from  the  respec¬ 
tive  subsystem  specialists.  The  triangular 
distribution  methodology  (discussed  and 
demonstrated  in  Section  III  A  2)  was  used  to 
generate  the  expected  value  (mean)  and  esti¬ 
mated  uncertainty  (i.e.,  standard  deviation, 
o)  for  each  of  the  subsystem  element.*. 
Parametric  cost  models  were  used  to  deter¬ 
mine  the  subsystem  element  costs.  The 
standard  deviations  of  the  weight  estimates 
were  combined  with  the  standard  error  of 
estimate  of  the  CERs  by  means  of  the 
parametric  error  propagation  equation  (5), 
thereby  providing  an  estimate  of  the 
standard  deviation  (uncertainty)  in  the  sys¬ 
tem  element  cost.  Tables  5  and  6  3how  the 
subsystem  costs  by  cost  element  along  with 
the  standard  deviation  for  the  non-recurring 
costs  and  recurring  costs,  respectively. 
The  coefficients  of  variation  or  fractional 
standard  deviations  are  also  shown  as  a 
percentage.  The  standard  deviations  are 
aggregated  by  summation  into  a  total  system 
standard  deviation.  The  standard  deviations 
that  would  be  obtained  if  RSS  techniques 
were  used  are  shown  for  comparison. 

The  total  system  acquisition  cost,  not 
including  launch  or  operations  and  support 
costs,  is  shown  in  Table  7,  along  with  Che 
aggregated  cost  uncertainty  as  reflected  by 
the  standard  deviation. 


by  Dr.  R.  H.  Huddlestone  and  presented 
jace  Corporation,  Interoffice  Correspon- 
distribution. ) 


Table  5.  Aggregation  of  System  Non-Recurring  Costs 
and  Their  Uncertainties 


Non-Recurring  Cost  in  FY  B2  Dollars 


Parametric 

Estimated 

Coefficient 

Cost  Est. 

Std.  Dev. 

of  Variation 

System  Element 

Mt 

O  Mi 

Z 

Mission  Equip.  (M.E.) 

Element  A 

50.0 

3.0 

6.0 

Element  B 

225.4 

33.2 

14.7 

Element  C 

36.3 

2.7 

7.4 

M.E.  Subtotal 

311.7 

38.9 

C33.4  RSS) 

12.5 

(10. 7Z  RSS) 

Spacecraft  (P/C) 

Structure  &  Mech. 

12.2 

2.1 

17.2 

Thermal  Control 

3.6 

2.1 

58.3 

Electric  Power 

14.9 

2.9 

19.5 

Trk. ,  Tel.  &  Coram'd 

53.1 

2.6 

4.9 

Stab.  &  Control 

19.0 

7.8 

41.1 

Aux.  Propulsion 

3.0 

0.9 

30.0 

S/C  Subtotal 

105.8 

18.4 

(9.3  RSS) 

17.4 

(8.8Z  RSS) 

Integration 

15.6 

1.9 

12.2 

Qual.  &  Space  Proto. 

472. 1 

53.4 

11.3 

Ground  Support  Equip. 

9.6 

1.0 

10.4 

Fee 

64.0 

8.0 

12.5 

Total  Non-Recurring 


978.8 


121.6 

(64.2  RSS) 


12.4 

(6.6X  RSS) 


Table  6.  Aggregation  of  System  Recurring  Costs 
and  Their  Uncertainties 


Non-Recurring  Cost  in  FY  32  Dollars 


, 

Parametric 

Estimated 

Coefficient 

Cost  Est. 

Std.  Dev. 

of  Variation 

System  Element 

Mfc 

0  M& 

X 

Mission  Equip.  (M.E.) 

Element  A 

36.3 

3.8 

10.5 

Element  B 

269.4 

23.4 

10.5 

Element  C 

63.6 

6.4 

10.1 

M.E.  Subtotal 

369.3 

38.6 

10.5 

(29.4  RSS) 

(8.0Z  RSS) 

Spacecraft  (S/C) 

Structure  &  Mech. 

19.2 

3.6 

18.3 

Thermal  Control 

4.8 

3.4 

70.8 

Electric  Power 

66.6 

3.3 

12.5 

Trk. ,  Tel.  &  Comm'd 

95.4 

6.3 

6.6 

Stab.  &  Control 

34.8 

7.2 

20.7 

Aux.  Propulsion 

6.6 

0.3 

4.5 

S/C  Subtotal 

227.4 

29.1 

12.3 

9 

(13.6  RSS) 

(6.0Z  RSS) 

Integration 

18.6 

1.9 

10.2 

Launch  Site  Support 

8.5 

1.1 

12.9 

Fee 

43.7 

4.9 

11.2 

Total  Recurring 

667.5 

75.6 

11.3 

(32.8  RSS) 

(4.9X  RSS) 

Table  7. 

System  Acquisition 
(In  FT 

Cost  and  Cost  Uncertainty  (a) 

82  Dollars) 

Estimated 

Estimated 

Coefficient 

Procurement 

Cost 

Std.  Dev. 

of  Variation 

Phase 

M  $ 

a  M  S 

X 

Non-Recurring 

973.8 

121.6 

12.4 

Recurring  (6) 

667.5 

75.6 

11.3 

Total 

1646.3 

197.2 

12.0 

* 
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V.  COST-PROBABILITY  (COST  RISK)  RELATIONSHIP 


The  objective  of  system  cost  risk 
analysis  is  to  portray  the  financial 
resources  required  to  acquire  a  given  system 
in  a  concise  yet  realistic  manner.  A  point 
cost  estimate,  altnough  concise,  does  not 
realistically  represent  the  actual  uncer¬ 
tainties  that  prevail.  A  cost-probability 
curve  for.  Che  given  system  can  depict  the 
measure  of  cost  risk  associated  with  the 
acquisition  of  Che  system. 

A.  METHODOLOGY  FOR  GENERATING  A  COST- 

PROBABILITY  CURVE 

The  expected  (mean)  total  system  cost 
and  its  standard  deviation  -  whose  genera¬ 
tion  was  described  and  illustrated  in  Che 
previous  section  -  are  sufficient  for  Che 
development  of  a  cost-probability  curve. 
The  expected  cost  corresponds  to  a  0.5  prob¬ 
ability.  If  a  normal  probability  distribu¬ 
tion  is  assumed,  reference  Co  cumulative 
probability  tables  shows  chat  the  expected 
cost,  +  1  standard  deviation,  corresponds  to 
a  0.841  cumulative  probability.  This  means 
that  the  probability  i-s  0.841  or  84. 1Z  chat 
Che  cose  will  be  less  Chan  Che  amount 
corresponding  Co  Che  expected  cost,  +  1 

standard  deviation.  The  expected  cost  plus 
twice  Che  standard  deviation  corresponds  Co 
a  cumulative  probability  of  0.977  or  97. 7Z. 
Similarly,  Che  expected  cose,  -  1  standard 
deviation,  corresponds  to  a  0.159  probabil¬ 
ity  and  the  expected  cost,  -  2  standard 

deviations  corresponds  to  a  0.023  probabil¬ 
ity.  The  cost  probability  curve  can  be 
generated  by  plotting  Che  above  points  for 
cost  versus  probability  on  standard  prob¬ 
ability  paper  and  connecting  them  with  a 
straight  line. 

The  three  prerequisites  for  generating  a 
cost-probability  curve  are: 


1.  The  availability  of  Che  expected 
(mean)  cost  with  its  standard 
deviation. 

2.  The  assumption  of  normality  (for  at 
least  the  upper  tail  of  Che  dis¬ 
tribution,  as  discussed  in  Section  C 
below) . 

3.  The  following  table  of  cumulative 
probabilities  versus  standard 
deviations  (a): 


Cumulative 
Probability  Z 


Expected  Cost 

20 

0.0228 

2.28 

Expected  Cost 

- 

10 

0.1587 

15.87 

Expected  Cost 

(mean) 

0.5000 

50.00 

Expected  Cost 

+ 

10 

0.8413 

84.13 

Expected  Cost 

+ 

20 

0.9772 

97.72 

B.  EXAMPLE  OF  COST-PROBABILITY  RELATIONSHIP 

DETERMINATION 

A  cost-probability  relationship  will  be 
developed  for  the  total  system  cost  fcl646.3 
million,  and  standard  deviation  [>197.2  mil¬ 
lion  developed  in  Section  IV  D  and  shown  in 
Table  7.  The  cost  probability  Curve  can  be 
developed  by  plotting  Che  following  cost 
versus  percent  probability  values  on  prob¬ 
ability  paper,  as  illustrated  in  Figure  5: 


Cc8t 

Z  Prob. 

Expected  Cost 

Expected  Cost  +  lo: 

1646.3 

50.0 

[1646.3  +  197.2  =] 

1843.5 

84.1 

Expected  Cost  -  lo: 

[1646.3  -  197.2  -] 

1449.1 

15.9 

Expected  Cost  +  20: 

[1646.3  +  2(197.2)  =] 

2040.7 

97.7 

Expected  Cost  -  2o: 

[1646.3  -  2(197.2)  -] 

1251.9 

2.3 
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The  steeper  the  slope  of  the  curve 
(i.e.,  relatively  narrow  cost  range  about 
the  expected  cost),  the  less  the  cost  risk; 
the  shallower  the  slope  of  the  curve  (i.e., 
relatively  wide/broad  cost  range  about  the 
expected  cost),  the  greater  the  cost  risk. 

C.  TRUNCATION  RATIONALE 

The  two  basic  reasons  why  system  costs 
are  estimated  are  (1)  to  aid  in  determining 
whether  the  anticipated  benefits  warrant  the 
expenditures  necessary  to  acquire  the  sys¬ 
tem,  and  (2)  to  determine  the  funding  that 
will  be  required  for  its  acquisition.  A  key 
characteristic  of  most  advanced  systems  is 
their  relatively  large  cost  -  developmental, 
production,  and  operations.  The  funding  of 
a  system  indicates  that  the  anticipated 
benefits  are  judged  to  be  substantially 
larger  than  its  costs,  or  else  its  develop¬ 
ment  would  be  irrational.  The  premature 
loss  of  the  benefits  of  a  key  system  could 
have  calamitous  consequences.  Thus,  program 
managers  responsible  for  the  acquisition  of 
systems  have  to  balance  the  dire  conse¬ 
quences  of  diminished  performance  or  pre¬ 
mature  failure  against  the  expenditures 
incurred  in  system  acquisition  and,  as 
discussed  in  Sections  IV  B  1  and  IV  B  2, 
would  likely  opt  for  enhancing  the  weakest 
link(s)  rather  than  underrunning.  Conse¬ 
quently,  the  expected  cost  (0.5  probability) 
constitutes  a  floor  on  the  system  cost; 
therefore,  it  is  recommended  Chat  the 
cost-probability  curve  be  viewed  as  credible 
only  upward  from  the  expected  cost  (0.5 
probability  point). 

D.  FUNDING  LEVEL  (RISK-ADJUSTED  COST 

ESTIMATE:  R-ACE)  DETERMINATION 

Although  the  cost-probability  curve  is  a 
more  valid  portrayal  of  the  prevailing 


reality  than  a  point  cost  estimate,  the 
budgetary  process  is  incapable  of  treating  a 
probabilistic  spectrum  of  costs.  Therefore, 
the  probabilistic  costs  depicted  on  the  cost 
curve  need  to  be  translated  into  a  meaning¬ 
ful  fixed  value  for  budgetary  requirements. 
A  basic  guideline  for  accomplishing  this  is 
to  use  the  cost  associated  with  a  probabil¬ 
ity  that  reflects  the  level  of  novelty 
inherent  in  the  system  -  the  risk-adjusted 
cost  estimate  (R-ACE).  For  example,  the 
cost  corresponding  to  a  0.6  to  0.7  probabil¬ 
ity  level  could  be  used  for  systems  with  a 
substantial  legacy  from  prior  systems  versus 
a  cost  corresponding  to  a  0.8  to  0.9  prob¬ 
ability  level  for  systems  incorporating  a 
substantial  technological  advancement. 
Major  architectural  and  engineering  firms 
tend  to  favor  a  0.85  probability  level  for 
cost  risk  estimates  of  new  major  construc¬ 
tion  projects. 

The  source  of  the  cost  increase  being 
compensated  for  by  the  R-ACE  is  usually  some 
aspect  of  actual  or  potential  technological 
def iciency(s) .  The  presence  of  a  tech¬ 
nological  deficiency  is  often  referred  to  as 
"technical  risk."  Major  categories  of  tech¬ 
nical  risks  are  those  that  are  associated 
with  both  the  RDT&E  phase  of  a  system  as 
well  as  with  its  production  phase.  A 
prerequisite  to  the  initiation  of  a  success¬ 
ful  RDT&E.  phase  (or  its  cost  estimation)  is 
that  the  physical  laws  and  principles  on 
which  the  functioning  of  the  system  will 
rely,  mus t  be  in  hand.  It  is  foolhardy  to 
even  attempt  to  estimate  the  cost  of 
achieving  a  technical  breakthrough.  Any 
cost  estimate  of  a  system  for  which  the 
basic  technology  i3  not  in  hand  is  meaning¬ 
less.  The  history  of  the  nuclear  powered 
aircraft  program  is  brought  to  mind. 
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The  technical  risks  in  the  RDT&E  phase 
are  usually  associated  with  the  efficient 
and  reliable  implementation  .of  known 
physical  laws  and  principles  through 
engineering  design.  Even  then,  substantial 
risks  are  occasionally  encountered  as  in  the 
case  of  the'  success-eluding  program  for 
developing  an  active  refrigerator  for  cool¬ 
ing  spacecraft  payload  sensors  (cryo- 
cooler) .  A  nominal  measure  of  the  impact  of 
the  technology  implementation  risk  is 
recognized  as  being  incorporated  within  the 
historically-derived  subsystem  RDT&E  CERs 
and  their  standard  errors  of  estimate  that 
constitute  the  cost  estimating  data  base. 

The  second  area  of  potential  tech¬ 
nological  deficiencies  that  create  technical 
risk  is  associated  with  production, 
specifically  that  stemming  from  a  lack  of 
manufacturing  or  testing  know-how  or  both. 
For  example,  it  is  one  thing  to  develop  and 
produce  a  demonstration-of-principle  mosaic 
focal  plane  containing  16  or  even  200 
detectors  under  laboratory  conditions;  it  is 
quite  another  to  mass  produce  an  operational 
version  with  say  10  million  detectors.  (The 
"development"  of  the  production  facilities 
may  present  an  even  greater  technical 
challenge  chan  the  actual  development  of  the 
detector.) 

The  resolution  of  technical  risk  (i.e., 
the  elimination  of  technological  deficien¬ 
cies)  i3  accomplished  through  the  acquisi¬ 
tion  of  new  knowledge,  which  may  be  viewed 
as  a  function  of  time  (schedule)  and  funding 
(cost).  Technical  risk  per  se  has  no  unique 
intrinsic  substance;  it  manifests  itself 
through  some  combination  of  cost  and  sched¬ 
ule  risk  -  cost,  through  the  focusing  of 
additional  skilled  scientific  and  engineer¬ 
ing  talent  on  the  surmounting  or  elimination 


of  the  impeding  deficiencies;  schedule 
through  the  application  of  the  scientific 
and  engineering  talent  over  a  longer  period 
of  time  or  more  intensely,  which  again  is 
basically  a  cost  impact.  (Either  accelera¬ 
tion  or  stretch-out  of  a  program  from  a 
normalized,  baselined  schedule  usually 
results  in  increased  costs  because  of 
increased  labor  requirements.)  Co3t 
impacts,  although  they  capture  the  major 
essence  of  schedule  changes,  omit  the 
initial  operational  capability  (IOC)  consid¬ 
eration  or  program  criterion.  Nevertheless, 
cost  impacts  capture  most  of  the  essence  of 
schedule  risk.  Thus,  cost  impacts  con¬ 
stitute  an  acceptable  surrogate  for  tech¬ 
nical  risk.  This  is  borne  out  in  the  testi¬ 
mony  given  by  Deputy  Secretary  of  Defense, 

F.  C.  Carlucci,  on  9  February  1982  before 

13 

the  House  Committee  on  Armed  Services  in 
which  he  reported  the  increases  in  the  FY  83 
DOD  budget  for  the  technical  risks  associ¬ 
ated  with  the  RDT&E  phase  of  12  major 
programs.  The  average  percent  funding 
increase  for  “technical  risk"  for  the  12 
programs  was  21.12.  (The  median  was  between 
17.7  and  21. 1Z ) .  Thus,  in  the  absence  of 
other  data  on  which  to  establish  a  R-ACE,  a 
cost  increase  on  the  order  of  20Z  may  be  the 
best  estimate  possible  under  the  circum¬ 
stances.  However,  experience  gained  from 
the  application  of  the  R-ACE  technique  would 
soon  provide  guidelines  for  selecting  prob¬ 
ability  levels  (and  hence  costs)  commensur¬ 
ate  with  the  inherent  uncertainties,  nature, 
and  characteristics  of  the  system  being 
costed . 

The  difference  between  the  cost  corres¬ 
ponding  to  the  selected  risk  probability 
level  and  the  expected  cost  should  go  into  a 
program  reserve,  possibly  such  as  that 
recommended  in  the  Total  Risk  Assessing  Cost 
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Estimate  (TRACE)  concept.  Whereas,  the 
budgetary  funding  request  would  correspond 
to  the  cost  associated  with  the  estimated 
risk  probability  level,  the  procurement  con¬ 
tract  and  any  incentive/award  fees  should  be 
based  on  the  proposed,  "expected"  cost. 
Were  the  contract  award  to  be  made  at  say 
the  0.85  probability  cost  level,  the  mani¬ 
fold  pressures  prevailing  on  the  program 
management  to  produce  a  reliable,  high 
quality  system  would  assure  that  the  incre¬ 
mental  (reserve)  funds  would  be  committed  to 
eliminating  weak  links  and  enhancing  the 

system,  thereby  raising  the  probability  of 

insufficient  funds  being  available  to  over¬ 
come  any  unexpected  adversities  late  in  the 
program.  The  likely  net  result  would  be  an 
overrun  of  the  R-ACE  amount.  To  preclude 
this,  the  reserve  should  be  in  the  tight 
control  of  the  system  program  office  or,  in 
the  case  of  major  system  contracts,  in  the 
control  of  a  top  level  DOD  service  board, 

but  not  in  the  control  of  the  contractor, 

who  should  endeavor  to  meet  his  initial  C03t 
estimate. 

VI.  ADDITIONAL  CONSIDERATIONS  IN 

APPLICATION  OF  METHODOLOCY 

A.  INFLATION  UNCERTAINTY 

Although  the  cost  risk  methodology 
presented  ha3  described  and  demonstrated 
(1)  techniques  for  estimating  the  cost 
uncertainty  associated  with  the  various  cost 
elements,  (2)  a  recommended  methodology  for 
the  aggregation  of  the  cost  uncertainties, 
(3)  a  methodology  for  the  generation  of 
cost-probability  curves,  and  (4)  some 
initial  guidelines  for  the  selection  of  a 
risk-adjusted  cost  estimate,  consideration 
of  inflation  has  been  excluded  to  this  point 
for  clarity  of  presentation.  All  treatment 
of  costs  and  cost  uncertainties  has  been  in 


base-year  dollars.  Cost  estimates  for 
budgetary  requirements  generally  need  to 
reflect  the  impacts  of  inflation  and  be 
expressed  in  then-year  dollars.  This  can  be 
accomplished  as  described  below. 

The  RDT&E  and  production  costs  along 
with  their  uncertainties  (i.e.,  standard 
deviations)  can  be  spread  by  the  use  of 
historically  derived  spread  factors, 
programmatic  schedules,  or  by  other  means, 
over  the  years  in  which  they  are  expected  to 
be  incurred.  To  dj  this,  the  prorated,  or 
allocated  cost  estimate  as  well  as  its 
uncertainty  estimate  for  a  given  year  can  be 
generated  by  multiplying  the  total  RDT&E  or 
production  costs  and  its  cost  uncertainty 
a  by  the  appropriate  spread  factor.  The 
prorated  cost3  together  with  the  cost 
uncertainties  thus  generated  for  each  future 
year  can  then  be  multiplied  by  the  expected 
inflation  factor  (e.g.,  Office  of  the 
Secretary  of  Defense  (OSD)  inflation  factor) 
for  that  year.  This  procedure  corresponds 
to  that  described  under  the  constant  multi¬ 
plier  approach  in  Section  III  E  3.  Thus 
spread  (allocated)  and  inflated  cost  esti¬ 
mates  as  well  as  their  uncertainties  can  be 
generated  by  direct  multiplication  by  the 
spread  factors  and  inflation  factors. 

B.  ALTERNATIVE  AGGREGATIONS  AND  COST- 

PROBABILITY  PLOTS 

Once  the  costs  and  their  uncertainty 
estimates  have  been  spread  across  the 
corresponding  years  and  inflated,  the  final 
steps  are  (1)  to  aggregate  the  costs  and 
their  uncertainties  (by  adding  the  standard 
deviations),  (2)  generate  coot-probability 
plots,  and  (3)  select  R-ACE  values.  The 
direction  of  aggregation  and  the  resulting 
plots  depend  on  the  cost  estimates  that  are 
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being  sought.  For  example,  if  annual  fund¬ 
ing  requirements  are  desired,  the  aggrega¬ 
tion  will  be  by  year;  if,  on  the  other  hand, 
funding  by  program  phase  is  desired, 
aggregation  will  be  by  RDT&E  phase,  produc¬ 
tion  phase,  etc.;  if  total  program  cosc  is 
desired,  all  costs  as  well  as  their  associ¬ 
ated  uncertainties  need  to  be  aggregated. 
After  the  sought  costs,  for  example  annual 
funding  requirements  and  their  uncertain¬ 
ties,  are  totaled  by  year,  then  cost- 
probability  curves  can  be  generated  by  year 
and  provided  to  the  contracting  agency  for 
tt-ACE  selection  of  funding  requests. 

VII.  SUMMARY 

The  over-all  methodology  presented  for 
determining  total  system  cost  risk  is  judged 
to  constitute  a  significant  advance  in  the 
area  of  cost  risk  determination.  Although 
it  admittedly  favors  pragmatic  results  over 
"noncost-effective"  statistical  rigor,  it  is 
basically  sound.  The  fundamental  value  of 
the  methodology  is  that  it  recognizes 
explicitly  and  derives  analytically  impacts 
on  cost  of  the  uncertainties  prevailing  in 
both  system  element  estimates  and  in  cost 
estimating  relationships  -  uncertainties 
that  were  formerly  ignored,  but  which  con¬ 
tribute  significantly  to  cost  estimating 
"error."  Methodologies  are  presented  for 
arriving  at  the  uncertainty  surrounding 
system  element  costs  that  are  derived  by 
alternative  costing  approaches  (e.g.,  para¬ 
metric,  engineering,  analogy,  and  constant 
multiplier) . 

It  is  shown  that  the  uncertainties  in 
the  costs  of  the  system  elements  cannot  be 
validly  aggregated  by  the  conventional  tech¬ 
nique  of  adding  the  variances  (root-^um- 
squaring).  The  proposed  approach  -  which  is 


recognized  as  being  conservative  in  that  the 
errors  are  not  diminished  by  the  process  of 
error  aggregation  -  consists  of  adding  the 
standard  deviations  associated  with  the 
costs  of  the  system  elements.  Pragmatic¬ 
ally,  the  estimate  of  the  total  cost  error 
tends  to  be  "reasonable."  By  generating  a 
cost-probability  curve  (truncated),  the  cost 
risk  involved  can  be  perceived.  However, 
for  budgetary  purposes,  a  risk-adjusted  cost 
estimate  (R-ACE),  corresponding  to  a  prob¬ 
ability  level  commensurate  with  the  novelty 
of  the  system,  needs  to  be  selected.  The 
risk-adjusted  cost  thus  selected  should  be 
used  for  the  determination  of  budgetary 
requirements.  A  management  reserve  base  on 
the  difference  between  the  R-ACE  and  the 
expected  cost  (0.5  probability)  should  be 
established  inasmuch  as  the  expected  cost  is 
virtually  certain  to  be  exceeded. 

By  compiling  data  on  the  results  of  the 
application  of  this  methodology,  particular¬ 
ly  on  the  resultant  accuracy  of  the  selected 
R-ACE  values,  specific  guidelines  or  rela¬ 
tionships  for  selecting  probabilities  com¬ 
mensurate  with  the  novelty  of  systems  can  be 
established.  Complex,  first  generation 
systems  might  necessitate  a  R-ACE  at  the  0.9 
probability  level,  whereas  fifth  generation 
systems  might  be  accurately  estimated  at  the 
0.7  probability  level. 

Identification  of  such  probability- 
determining  guidelines  could  further  improve 
the  accuracy  of  system  cost  estimating. 

The  presentation  of  this  methodology  is 
but  the  first  step.  Its  application 
together  with  documentation  of  results,  will 
permit  the  identification  and  correction  of 
any  deficiencies.  Toward  this  end,  the 
methodology  developed  for  propagating  error 
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VIII.  FUTURE  EFFORT 


(uncertainty)  in  the  parametric  costing 
approach  together  with  the  methodology  for 
aggregating  the  .cost  uncertainty  have  been 
incorporated  into  the  Air  Force  Space 
Division  computerized  version  of  the 
Unmanned  Spacecraft  Cost  Model.* 

The  potential  applicability  of  this 
methodology  or  its  elements  extends  beyond 
cost  risk  estimation  per  se.  The  method¬ 
ology  for  the  identification  of  errors, 
their  propagation  through  mathematical 
relationships  and  ultimate  aggregation 
appears  to  be  applicable  to  technical  fields 
other  than  cost  estimating. 

By  continuing  identification,  reduction, 
and  eventually  elimination  of  sources  of 
cost  estimating  error,  more  accurate  cost 
estimates  will  be  achieved.  The  methodology 
developed  and  described  herein  is  a  step  in 
that  direction.  Consistently  accurate 
system  cost  estimates,  however,  should 
realistically  be  viewed  as  an  elusive  goal 
because  of  the  inevitable  presence  of 
inherently  random  or  uncontrollable  events 
that  impact  costs.  (Even  with  complete 
estimating  accuracy  at  all  steps,  a  0.9 
probability  level  cost  will  be  exceeded  10Z 
of  the  time  because  of  random  influences.) 
Nevertheless,  significant,  further  improve¬ 
ments  can  be  achieved  through  better  under¬ 
standing  of  the  sources  of  cost  errors  and 
the  decisions  that  influence  the  procurement 
of  the  system  being  costed. 

This  paper  wa3  initially  published  in  an 
Aerospace  Corporation  Technical  Report 
( SD-TR-83-44 ,  dated  3  June  1983). 


In  examining  areas  of  deficiencies  in 
cost  analysis,  two  appear  to  be  outstanding 
and  so  constitute  prime  candidates  for 
future  effort.  The  first  is  the  replacement 
of  the  method  of  least  squares  by  the  method 
of  least  distance  for  generating  coat  esti¬ 
mating  relationships.  When  relationships 
developed  by  the  method  of  least  squares  are 
used  to  predict  costs  in  the  upper  ranges  of 
the  independent  variable,  the  results  are 
significant  underestimates.  The  method  of 
least  distance,  which  will  be  described  and 
illustrated  in  the  subsequent  effort,  does 
not  possess  this  deficiency. 

The  second  area  of  deficiency  is  in  the 
estimation  and  quantification  of  key  aspects 
of  schedule  risk.  Results  of  progress  on 
these  two  efforts  will  be  reported  at  the 
meetings  of  the  Space  Systems  Cost  Analysis 
Group  (SSCAG). 


*  Jose  Gutierrez  of  The  Aerospace  Corporation  was  responsible  for  modifying  the  cost  model  (on 
the  Hewlett-Packard  9845)  such  that  both  numerical  results  and  cost-probability  plots  would 
be  generated. 
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Oyer  the  past  decade,  research  and  synthesis 
findings  on  decision  making  have  led  to  the 
.unequivocal  conclusion  that  earlier  concept- 
(ji plications  about  the  decision  process  were 
either  overly  simplistic  or  lacking  in  veri- 
y^Tjicality.  The  nature  of  the.  decision  task, 
^nd  the  conditions  under  which  it  is  performed 

Oave  a  profound  influence  on  the  decision 

rocess.  These  effects  include  the  decision 
^maker 1 s  view  of  the  process,  and  (probably) 

»  ^information  processing  strategies  which  the 
_  decision  maker  may  be  unaware  of  having 
^^jchosen.  Further,  there  is  a  growing  body  of 
literature  which  suggests  that  the  nature  of 
* -» «*the  decision  process  is  strongly  influenced 
_  ?Lv  the  organizational  level  at  which  the 
*"**»decision  maker  is  located,  i.e.,  what  his 
critical  functions  in  the  organization  are, 
and  the  nature  of  the  cognitive  skills  he 
therefore  must  bring  to  the  task.  The  purpose 
of  the  present  paper  is  to  present  some  of 
these  data,  together  with  some  possible  impli- 
j  cations  they  have  for  decision  making  under 
conditions  of  uncertainty,  and  for  risk 
management .  - - - 


These  latter  considerations  will  have  partic¬ 
ular  relevance  for  program  managers,  for  the 
following  reasons.  If  we  assume  that  program 
managers  are  differentially  effective,  then 
it  is  reasonable  to  assume  that  part  of  the 
variance  in  effectiveness  is  ad  hominum  and 
part  is  environmentally  determined.  To  some 
extent,  ad  hominum  variance  can  be  controlled 
through  individual  assessment  and  selection, 
though  this  is  an  expensive  venture.  However, 
environmental  engineering  offers  much  more 
promise.  It  seems  highly  likely  that  environ¬ 
ments  can  be  made  more  adzptive  for  effective 
decision  performance  w ith  relatively  little 
cost  to  organizations.  If  this  is  a  correct 
assumption,  then  interesting  new  approaches 
to  improving  decision  performance  under 
conditions  of  uncertainty  will  become  avail¬ 
able  . 

First,  the  following  definitions  of  risk  and 
uncertainty  seem  useful  (Lopes,  1983): 

•  Risk  defines  the  condition  under  which 
the  possible  outcomes  of  &  decision  are 
known,  and  so  are  the  probabilities 
attached  to  each. 

•  Uncertainty  defines  the  condition  under 
which  the  outcomes  are  known  but  the 
probabilities  are  not.  (Further,  there 
may  be  conditions  of  extreme  uncertainty 
under  which  not  all  the  outcomes  are 
either  knowable,  or  anticipated  if 
knowable. ) 


The  essence  of  this  distinction  is  between  a 
condition  of  rationality  or  one  of,  at  best, 
bounded  rationality.  The  problem  is  that  a 
substantial  amount  of  the  early  work  on  dec¬ 
ision  making,  and  many  of  the  early  concept¬ 
ualizations,  addressed  risk  to  a  far  greater 
extent  than  uncertainty,  as  thus  defined. 
However,  most  of  the  decision  making  done  by 
humans  at  senior  levels  of  responsibility, 
either  program  managers  or  executives,  is  not 
done  under  conditions  of  risk,  as  defined,  but 
rather  under  conditions  of  uncertainty. 

Three  useful  reviev  s  of  the  decision  literature 
have  appeared  during  the  past  six  years 
(Slovic,  Fischoff,  and  Lichtenstein,  1977; 
Einhom  and  Hogarth,  1981;  Payne  1982).  In  the 
first  of  these,  Slovic,  Fischoff,  and 
Lichtenstein  included  a  substantial  treatment 
of  decision  aiding,  but  at  the  same  time  noted 
some  emerging  general  findings  that  challenge 
rational  decision  theories,  i.e.,  those  which 
have  as  a  basic  assumption  maximization  of 
expected  value  (EV),  expected  utility  (EU),  or 
subjective  expected  utility  (SEU).  First,  in- 
•  formation  load  consistently  has  shown  strong 
impact  on  information  acquisition  utilization 
strategies.  Second,  strategies  for  search 
and  evaluation  may  vary  from  one  stage  of  a 
decision  problem  to  another,  a  finding  which 
repeatedly  occurs  and  which  confounds  ration¬ 
ality  assumptions.  Payne's  review  adds  a 
third  general  finding:  decision  makers  seem 
to  have  some  kinds  of  learned  rules  which  are 
probably  context  bound  and  generalized  across 
situations  (that  is,  they  are  ways  of  dealing 
with  decision  requirements)  whether  it  is 
appropriate  to  generalize  them  or  not.  Payne 
noted  that  Abelson's  scripts  and  Pitz's  pro¬ 
duction  systems  are  good  examples. 

The  second  of  the  three  reviews  cited  earlier 
had  a  broader  objective: 

"...to  place  behavioral  decision  theory  in 
a  broader  perspective,  emphasizing  impor¬ 
tance  of  attention,  memory,  cognitive 
representation,  conflict,  learning,  and 
feedback." 

As  does  Payne,  Einhom  and  Hogarth  challenge 
normative  models: 

"Judgement  and  choice  are  strongly  in¬ 
fluenced  by  seemingly  minor  'changes  in 
task  and  context.  And  information  search 
and  evaluation  strategies  are  interdepend¬ 
ent.  A  variety  of  strategies  exists,  and 
little  is  known  about  criteria  for  rule 
taking,  rule  shifting,  and  the  choice  of 
evaluation  strategy." 
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Uncertainty  arises  from  the  environment,  from 
equivocal  cue-criterion  relationships,  incon¬ 
sistency  in  individual  information  combination 
strategies,  and  the  question  of  how  cues  should 
be  weighted  in  relation  to  their  predicitive- 
ness.  Finally,  they  note  that  there  is  limited 
learning  capacity  in  humans,  and  that  humans 
paradoxically  show  learned  confidence  in  judge¬ 
ment  despite  obvious  low  validity  of  judgement. 

While  hu.  ->n  decision  performance  even  in  simple 
conditions  is  subject  to  characteristic  biases 
and  errors,  a  good  case  can  be  made  that  per¬ 
formance  becomes  worsa  as  task  complexity  in¬ 
creases.  Payne  (1982)  focuses  on  the  impact 
of  task  complexity,  noting  that  as  complexity 
increases,  information  load  presumably  also 
increases,  and  several  predictable  impacts 
reliably  are  found: 

•  Compensatory  strategies  shift  to  con¬ 
junctive  or  elimination-by-aspects 
strategies . 

•  Response  variability  increases  and 
choice  quality  decreases. 

•  Decision  makers  rely  more  on  negative 
information  to  reduce  complexity  (with 
increases  in  time  pressure). 

•  Risk  propensity  is  reduced  by  a  more 
constrained  time  horizon. 

•  Decision  makers  tend  more  strongly  not 
to  transform  information  but  rather  to 
use  it  in  the  form  conveyed,  which 
reduces  cognitive  strain  but  does  little 
to  enhance  quality  of  decisions. 

In  addition,  decision  makers  are  systematic  in 
violation  of  some  rationality  assumptions, 
but  nonsystematic  in  violation  of  others.  For 
example,  there  is  apparently  systematic 
tendency  to  be  risk  aversive  for  gain  and  risk 
seeking  for  losses.  However,  Lopes  (1983) 
seeks  to  explain  this  systematic  violation  of 
rationality  assumptions  in  terms  of  psychologi¬ 
cal  variables  such  as  the  decision  maker's 
status  and  long  term  goals,  which  of  necessity 
are  idiosyncratic.  Payne  makes  a  similar  point, 
that  decision  makers  operate  in  terns  of  a 
psychologically  relevant  outcome  space  (problem 
space)  which  may  or  may  not  conform  either  to 
reality  or  to  the  outcome  space  of  other 
decision  makers. 

Finally,  Payne  notes  that  little  is  known  about 
how  decision  strategies  are  learned.  Heuristics 
are  widely  used,  and  the  evidence  suggests  that 
they  are  context-dependent.  However,  once 
learned,  they  can  be  very  enduring.  A  great 
deal  more  needs  to  be  learned  about  how  they 
are  acquired,  and  how  feedback  processes  operate 
to  mediate  their  retention  over  time. 

As  was  noted  by  Payne,  Braunstein  and  Carroll 
(1978),  decision  research  to  date  has  focused 
more  on  outcomes  than  on  processes.  In  order 
to  understand  decision  performance  better,  we 
need  better  data  on  how  decision  makers  acquire 


and  use  information,  the  information  that 
eventually  is  used,  and  the  conditions  under 
which  it  is  used.  If  decision  making  i3  viewed 
more  from  the  perspective  of  information 
processing  benavior,  focusing  on  search/acqui¬ 
sition,  evaluation,  and  feedback/learning , 
avenues  may  well  be  found  to  improve  decision 
performance  through  attention  to  environmental 
and  individual  factors  that  influence  these 
processes.  The  remainder  of  this  review  will 
focus  on  these  issues.  Three  broad  categories 
will  be  addressed:  (a)  the  information  pro¬ 
cessing  characteristics  of  the  human  decision 
maker,  (b)  the  nature  of  the  function  served 
by  decisions  within  the  organizational  context, 
and  (c)  the  form  taken  by  the  requisite  infor¬ 
mation  in  relation  to  the  decision  function 
served  and  the  cognitive  skills  of  the  decision 
maker. 

Broadbent,  among  many  others,  has  contributed 
substantially  to  an  understanding  of  the 
characteristics  of  human  decision  makers  in 
terms  of  the  basic  processes  involved.  In 
a  seminal  article,  he  (Broadbent,  1977) 
described  research  on  control  systems  which 
mediate  throughout  processes  in  human  informa¬ 
tion  processing.  Processing  tasks  were  defined 
in  two  ways : 

•  Closed  tasks  -  open-chain  sequences  which 
require  no  check  with  the  environment  for 
execution. 

•  Open  tasks  -  unpredictable  sequences  of 
actions  in  relation  to  unpredictable 
series  of  events,  which  requires  contin¬ 
uing  check  with  environment. 

In  a  series  of  experiments,  Broadbent  demonstra¬ 
ted  that  human  information  processing  occurs 
at  least  at  two  levels,  and  that  these  levels 
may  correspond  to  the  type  of  tasks  performed. 
One  control  system  operates  in  an  open  chain 
fashion,  accepting  input  and  producing  output, 
without  any  feedback  loops.  A  second  control 
system  can  be  visualized  as  an  integrating, 
feedback  operated  processor,  with  a  capacity 
for  both  rewriting  the  rules  used  by  the 
open  chain  processor,  and  for  providing  it  with 
inputs.  Broadbent  visualized  these  as  "lower" 
level  and  "upper"  level,  respectively.  A  key 
assertion  is  that  lower  level  processes  can 
operate  in  parallel,  as  long  as  they  do  not 
require  the  same  input  sensory  modality,  but 
that  upper  level  processes  cannot.  Perhaps  of 
even  greater  interest,  he  theorized  that  control 
of  the  total  information  processing  task  shifts 
from  one  level  to  another,  depending  on  context, 
task,  and  feedback. 

Awareness  of  differential  processes  such  as 
these  is  not  reflected  in  the  methodology  re¬ 
ported  in  much  of  the  current  literature.  Many 
findings  are  based  on  laboratory  tasks  which 
probably  elicit  use  of  lower  level  processors, 
while  others  are  based  on  tasks  requiring  upper 
level  processors.  Specifically,  repeated 
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responding  in  a  simple  lottery  could  conceivab¬ 
ly  be  accomplished  by  a  lower  level  processor. 

In  such  tasks,  it  would  not  be  surprising  to 
find  the  use  of  quite  different  search/acquisi¬ 
tion  and  evaluation  strategies  in  those 
which  activate  upper  level  processors.  Further, 
it  would  not  be  surprising  to  find  that  the 
switching  control  system  levels  is  contingent 
on  such  variables  as  information  load,  with 
upper  level  systems  operating  under  conditions 
permitting  the  longer  processing  times  probably 
required,  and  switching  processing  to  lower 
level  systems  when  available  processing  times 
are  inadequate  for  the  functioning  of  upper 
level  systems.  More  research  correlating 
control  systems  with  tasks  by  conditions  is 
needed  for  developing  understanding  of  human 
information  processing  characteristics  of 
the  nature  of  decision  tasks. 

.In  preparation  of  this  review,  no  research  was 
found  on  information  processing  requirements 
for  program  managers.  However,  there  is  a 
growing  body  of  knowledge  about  decision  and 
information  processing  behavior  of  senior 
executives.  If  it  can  be  assumed  that  similar 
behaviors  are  required  of  senior  executives 
and  managers  of  complex  programs,  conclusions 
can  be  drawn  from  analysis  of  that  literature. 

Two  interesting  recent  publications  are  rele¬ 
vant  (Kotter,  1982;  Draft  and  Lengle,  1983). 

The  first  is  one  of  a  growing  number  of  works 
in  which  the  on-the-job  behavior  of  senior 
managers  is  reported  and  analyzed.  The  second 
is  an  analysis  of  the  information-processing 
task  of  the  senior  executive.  In  it,  the 
authors  assert  that  the  managerial  task  is  to 
make  sense  out  of  complex  decision  making  - 
and  it  can  be  argued  that  this  is  a  central  part 
of  complex  decision  making  -  and  to  coordinate 
internal  activities  within  the  organization. 

They  then  advance  the  concept  of  information 
richness,  which  is  defined  as  the  information 
carrying  capacity  of  data. 

*  Situations  of  greater  uncertainty  (equiv¬ 
ocality)  require  data  of  greater  richness, 
and 

•  Media  can  b e  scaled  in  terms  of  their 
richness  into  five  categories  from  most 
to  least  rich:  face-to-face,  telephone, 
letter,  memorandum  and  computer  printout. 
(Scaling  is  accomplished  through  use  of 
the  variables  of  chsnnels  utilized,  feed¬ 
back  capability,  source  and  language 
used. ) 

In  essence,  simple  phenomena  :an  be  managed 
with  simple  rules,  and  with  information  of  low 
richness;  complex  phenomena  require  rich  infor¬ 
mation,  and  probably  yield  more  easily,  in 
general,  to  the  use  of  heuristics  than  to  more 
mechanical  computational  procedures.  But  Kctter 
showed  convincingly  that  the  phenomena  of 
concern  to  senior  executives* in  his  sample  were 
indeed  complex.  He  further  provides  evidence 


concerning  their  sources  of  information  that 
strongly  supports  the  Draft  and  Lengel  hypothe 
ses.  Senior  executives  obtain  most  of  their 
information  from  "rich"  sources.  Similar 
support  is  contained  in  Mintzberg's  (1973) 
observational  data  on  the  job  of  the  senior 
manager.  Senior  managers  make  little  use  of 
low  richness  sources,  and  the  extent  to  which 
they  do  so  decreases  with  increases  in  uncer¬ 
tainty.  Not  only  are  their  most  critical  tasks 
focused  on  fuzzy  problems,  but  also  some  of 
the  relevant  data  are  essentially  political 
(Brightman,  1978).  Draft  and  Lengle  concluded- 

•  The  best  form  for  an  organization  to 
take  is  one  which  addresses  its  infor¬ 
mation  processing  needs  best. 

•  Management  information  systems  designers 
lack  a  coherent  theory  of  manager  needs 
and  manager  behavior. 

To  this  might  be  added  the  need  for  decision 
making  models  relevant  to  executive  function 
under  such  conditions,  and  the  unique  informa¬ 
tion  processing  skills  ct  good  decision  makers. 

Streufert  (1970,  1981)  addresses  these  skills 
from  a  cognitive  complexity  perspective  in 
research  dealing  with  information  search  and 
the  impact  of  load  stress  in  complex  decision 
making.  His  research  shows  that  decision 
makers  at  senior  levels  perform  better  if  they 
engage  in  cognitive  processes  of  differentiation 
and  integration  of  their  information  dimensions, 
and  then  use  these  dimensions  in  their  subse¬ 
quent  decisions.  As  expected,  work  pressure/ 
laod  reduces  the  capacity  to  do  this.  However, 
individuals  who  characteristically  do  more 
differentiation  and  integration  under  light 
loads  are  less  impacted  by  load  increases. 
Further,  predispositions  to  multidimensionality 
are  trainable,  though  with  difficulty,  perhaps 
paralleling  Payne's  observation  about  problems 
in  the  learning  of  decision  strategies.  As  a 
final  point,  Struefert  notes  another  finding 
paralleling  decision  research  findings  on 
feedback  acquisition  and  use.  In  his  experi¬ 
ments,  he1 found  that  the  amount  of  information 
needed  to  change  a  decision  exceeded  that  needed 
to  make  one.  One  might  therefore  conclude  that 
unidimensional  information  processors  will  not 
discover  bad  decisions  as  effectively  as  do 
multidimensionals .  If  a  correct  inference, 
cognitive  complexity  then  relates  to  search/ 
acquisition  and  evaluation  in  complex  tasks, 
though  of  necessity  it  would  not  in  simple 
tasks  not  requiring  cognitive  complexity. 

Thus  far,  select  findings  on  the  characteristics 
of  human  decision  makers  and  the  relation  of 
cognitive  skills  to  information  requir  anents 
and  decision  outcomes  have  been  presented. 

A  final  issue  is  the  time  horizon  (perspective) 
of  the  decision  maker.  Jaques  (1976)  has 
developed  a  theory  of  organizational  structure 
which  identifies  levels  of  performance  require¬ 
ments  and  relates  them  both  to  the  time  frames 


within  which  action  must  be  planned  by  level. 

For  practical  purposes,  seven  levels  are  defined, 
as  shown  in  Figure  One .  They  can  be  broken  into 
three  more  generaL  sets,  strategic,  general 
management,  and  operational  execution.  If  these 
levels  are  cross-matrixed  with  Streufert’s  cog¬ 
nitive  complexity  categories,  and  if  successive¬ 
ly  more  complex  program  development  and  manage¬ 
ment  activities  are  located  logically  within 
the  matrix,  the  cell  entries  shown  in  Figure 
One  emerge.  These,  of  course,  are  only  theoret¬ 
ical'.  However,  if  the  theory  is  accurate,  there 
are  several  strong  implications.  First,  the 
information  processing  requirements  of  succes¬ 
sively  higher  levels  of  organization  (and 
management  of  successively  more  complex  pro¬ 
grams)  should  increase  in  volume  and  complex¬ 
ity.  A  diagonal  from  lower  left  to  upper 
right  in  the  matrix  coincides  with  the  direc¬ 
tion  of  increasing  information  load.  Second, 
if  Jaques  is  correct  about  the  critical 
functions  of  incumbents  by  level,  the  degree  of 
concreteness  of  required  information  and  the 
form  it  sho'-ld  take  change  from  level  to  level. 
More  complex  problem  situations,  as  shown  on 
Figure  One,  require  the  use  of  more  abstract 
information,  i.e.,  consolidated  rather  than 
individual  elements,  transformed  to  show 
trends  as  opposed  to  sums  second  derivatives 
rather  than  first  derivatives,  and  so  on. 

Finally,  executives  and  program  managers  at 
the  more  senior  levels  should  have  longer 
time  horizons  than  shorter,  and  should  have 
demonstrated  capacity  for  multidimensional 
integration  information  processing  skills. 

The  research  base  thus  far  available  is  not 
large,  and  very  substantial  effort  is  now 
being  applied  to  the  creation  of  executive 
development  technology,  using  the  logic  of 
the  above.  Findings  from  this  research  will 
enable  substantially  more  confidence  in  the 
drawing  of  conclusions  than  now  is  possible. 
However,  it  seems  clear  that  some  improvements 
in  organizational  performance  (and  probably 
program  management)  can  be  achieved.  One 
fruitful  direction  is  assessment  of  potential 
cXcv.uij.vea  and  program  managers,  if  cognitive 
complexity  and  the  capacity  for  functioning 
over  extended  time  periods  (10-20  years)  are 
requirements  at  the  most  senior  levels, 
assessment  programs  to  permit  early  identifica¬ 
tion  and  development  of  talent  would  seem  to 
be  a  good  investment.  Training  and  development 
activities  could  then  be  purposefully  tailored 
to  permit  career  management  against  known 
executive  skill  performance  requirements.  A 
second  fruitful  approach  is  engineering  of  the 
information  environment.  Given  that  load 
decreases  decision  quality  and  limits  the 
multidimensionality  of  senior  executive  per¬ 
formance,  organizationally  imposed  information 
loads,  to  the  extent  they  exceed  optimum  levels, 
impact  on  organizational1  performance.  However, 
in  bureaucracies,  there  frequently  is  little 
filtering  of  the  content  of  imposed  loads  in 
relation  to  organizational  priorities.  Further, 


rules  proliferate  in  bureacuracies ,  and 
performance  is  audited  by  a  variety  of  "seniors" 
to  include  one's  formal  boss.  However,  the 
perspectives  of  the  "audit"  frequently  vary 
from  one  auditor  to  another,  and  in  some 
cases  do  not  directiy  reflect  awareness  of 
organizational  priorities  and  the  requirements 
for  effective  organizational  performance.  In 
terms  of  level  of  performance,  an  axiom  might 
be  that  the  greater  the  number  of  rules,  the 
less  integration  the  incumbent  can  show,  and 
the  lower  the  quality  of  decision  outcomes 
will  be,  outside  some  boundary  condition. 

Another  might  be  that  the  greater  the  load, 
e.g.,  through  imposition  of  performance 
requirements  established  by  an  organizational 
rule-making  subsystem  (such  as  personnel),  the 
lower  will  be  the  integrative  quality  of 
decision  outcomes.  Finally,  organizational 
structure  probably  influences  decision  quality 
outcomes.  For  complex  decision  making  and  the 
processing  of  information  under  conditions  of 
substantial  uncertainty,  mechanistic  organiza¬ 
tions  (like  bureaucracies)  probably  impose 
upper  limits  on  the  quality  of  executive  (and 
program  manager)  performance. 

If  these  inferences  are  correct,  progress 
toward  higher  quality  decisions  and  higher 
organizational  performance  probably  can  be 
realized  through  selection  and  development 
of  early  talent;  systematic  review  of  rules 
and  procedures  to  ensure  simplicity  and  con¬ 
sistency  in  complex  environments  (in  simple 
environments  it  does  not  matter);  systematic 
auditing  of  the  information  loads  imposed  on 
critically  important  senior  executives  and 
program  managers  to  ensure  they  are  not 
burdened  by  requirements  generated  by  sub¬ 
optimizing  subelements;  and  tailoring  of 
organizational  structure  to  match  the  comp¬ 
lexity  and  time  frame  cr  the  level  of  perform¬ 
ance  required. 
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THE  CONTRIBUTION  OF  CARDINAL  UTILITY  TO  REAL  LIFE  DECISIONS 

R.  0.  SWALM 
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SYRACUSE  UNIVERSITY,  SYRACUSE,  NY  13210 


We  shall  start  this  discussion  of  cardinal 
utility  theory  in  general  and  measurement  of 
risk  attitudes  in  particular  by  limiting  our¬ 
selves  to  situations  in  which  the  construct 
of  economic  man  -  one  who  is  driven  solely  by 
financial  considerations  -  is  reasonably  va¬ 
lid.  A  fair  mount  of  research  has  been  done 
here.  We  will  now  go  on  to  discuss  risk  atti¬ 
tudes  in  situations  which  are  not  controlled 
by  economic  considerations.  Unfortunately, 
almost  no  research  exists  in  this  area. 

For  many  years,  most  writers  who  considered 
probabilistic  environments  assuned  that  deci¬ 
sions  would  be  made  based  on  expected  values. 
This  is  not  always  true.  For  example,  it 
was  assumed  that  having  to  choose  between 
a  sure  dollar  and  a  50-50  gamble  on  either 
five  dollars  or  nothinq,  one  would  choose  the 
gamble,  since  its  expected  value  is  $2.50. 
Experiments  show  t.iat  almost  all  people  would 
indeed  make  this  choice.  However,  with  choos¬ 
ing  between  a  sure  million  dollars  or  a  50-50 
gamble  on  five  million  or  nothing,  most  people 
opt  for  the  sure  million.  We  demonstrated 
the  truth  of  these  statements  in  our  talk.  The 
fact  that  the  gamble  would  lead  to  an  expected 
value  two  and  a  half  times  greater  than  the 
million  swayed  almost  none  of  our  audience. 

In  the  16th  century,  in  his  famous  St.  Peters¬ 
burg  paradox,  Oaniel  Bernouli  demonstrated 
that  in  at  least  one  case,  respondents  were 
unwilling  to  base  decisions  on  expected  val¬ 
ues.  Philosophies,  logisticians,  statisti¬ 
cians  and  economists  have  discussed  this 
paradox  for  years.  But  it  was  not  until 
1946,  when  VonNeunann  and  Morgenstern  pu¬ 
blished  their  monunental  Theory  of  Games 
and  Economic  Behavior,  that  a  formal  treat¬ 
ment  of  risk  attitudes  would  allow  decisions 
made  in  risky  situations  to  be  predicted. 
They  showed  that,  given  certain  reasonable 
assumptions,  decision  makers  will  optimize 
on  the  expected  value  of  what  the  authors 
defined  as  utility.  This  formed  the  basis 
of  cardinal  utility  theory,  and  enabled  us 
to  quantify  risk  attitudes.  It  also  led  to 
much  confusion.  A  nimber  of  people,  most 
notably  Allais,  constructed  counter  examples 
in  which  most  people  would  not  choose  the 
alternative  having  the  greatest  expected 
utility.  Many  took  the  attitude  that  these 
counter  examples  destroyed  utility  theory. 
Others,  including  myself,  felt  that  this 
enhanced  its  usefulness.  We  take  the  posi¬ 
tion  that  only  when  normative,  or  prescri¬ 
ptive,  theory  is  at’  odds  with  descriptive 
behavior  that  the  theory  is  useful  beyond 
its  ability  to  massage  one's  prejudices. 


An  example  of  such  use  occurred  in  a  study  re¬ 
ported  in  the  Harvard  Business  Review.  In 
this  study,  I  found  that  many  decision  makers, 
employed  by  very  large  companies,  would  not 
recommend  investing  in  a  bid  which  offered  a 
50-50  probability  of  a  net  present  worth  of 
either  a  gain  of  $200,000  or  a  loss  of 
$20,000. 

Since  such  companies  have  many  many  opportuni¬ 
ties  of  this  kind,  and  have  very  large  re¬ 
sources,  it  is  easy  to  show  that  it  would  have 
been  in  the  company's  interest  to  take  such 
ri sks . 

Members  of  the  audience  hypothesized  that  an 
explanation  of  the  difference  between  the 
choices  that  should  be  made  and  those  that 
were  made  lay  in  control  procedures  which  pe¬ 
nalized  one  recommending  a  course  of  action 
leading  to  a  $20,000  loss.  My  research  tends 
to  confirm  chi s  hypothesis.  For  several  of 
my  respondents  asked,  perceptively,  if  I  had 
not  asked  what  they  would  do  rather  than  what 
they  should  do.  When  I  told  them  yes,  I  had, 
they  replied  that  they  would  not  recommend 
the  bid,  even  though  they  were  aware  it 
would  be  to  their  company's  interest  to  do 
so.  When  I  was  discussing  this  in  Norway  I 
was  told  that  I  had  simply  rediscovered  and 
verified  an  old  Norwegian  proverb.  It  says 
"It  takes  ten  times  as  much  profit  to  get  a 
pat  on  the  back  as  it  does  loss  to  get  a 
kick  in.  the  tail ." 

The  question  was  then  raised  as  to  whether  or 
not  similar  risk  averse  attitudes  would  tend 
to  be  held  by  civil  servants.  Unfortunately, 
no  research  exists  to  settle  this  question, 
but  it  is  surely  an  important  one  to  ask. 
Likewise,  we  can  offer  no  research  findings  on 
the  possibility  of  changing  employee  risk 
attitudes.  I  do,  however,  have  evidence  that 
in  one  company,  lower  management  perceived 
upper  management  to  be  far  more  risk  averse 
than  it  really  turned  out  to  be,  and  that 
they  felt  that  the  proper  risk  attitude  for 
the  company,  and  therefore  themselves,  was 
far  less  risk  averse  than  they  actually  were. 

Because  of  the  active  participation  of  thf 
audience  in  discussing  these  matters,  we  hae 
to  conclude  after  a  brief  discussion  of 
partially  successful  study  Jerry  Selman  anr 
I  made  on  risk  attitudes  held  by  militar 
officers. 
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Most  risk  analyses  begin  by  considering  the 
behavior  of  the  lowest  and  most  detailed  level  ctf 
all  possible  events  that  can  be  identified  in  the 
system  under  study.  Next,  risk  estimates  are 
made  for"  each  event-consequence  relationship 
and  aggregated  upward  to  obtain  a  total  risk 
estimate  of  the  process  under  study.  The 
aggregation  process  propagates  the  errors 
contained  in  the  detailed  risk  analyses,  and  often 
results  in  risk  estimates  whose  error  ranges  are 
too  wide  to  provide  useful  information.  An 
alternative  approach  starts  at  the  highest  level 
of  the  prohlem,  and  identifies  the  crucial 
decisions,  decision  maters,  alternatives  and 
parameters.  Agreements  regarding  specific 
decisions  and  subsequent  actions  are  sought  at 
the  beginning,  and  /"what  if1*4  situations  and 
conflicts  are  identified  in  else  agreement  cannot 
be  reached.  Thc^what  if*  situations  provide  the 
framework  for  more  detailed  and  focused  studies 
in  critical  areas.  A  variety  of  analyses,  such  as  a 
localized  version  of  the  bottom  up  risk  analysis 
approach  and  sensitivity  analysis,  focus  on  these 
open  ended  cases  to  resolve  them.  Unresolvable 
decision  conflicts  indude  value  judgnents  which 
risk  analysis  cannot  solve;  however,  by  malcng 
these  conflicts  viable,  the  focus  on  differences 
such  as  these  can  often  force  resolution  at  a 
higher  ,  management  level. 

Definitions 

Several  definitions  should  be  provided  as  a  basis 
for  discussion: 

Risk:  The  downside  of  a  gamble;  the  potential 
for  harm. 

Bottom-up  Risk  Analysis:  Taking  each  event 
that  can  occur  in  a  system  and  analyzing  the 
pathways  ieading  to  the  range  of  possible 
consequences,  and  aggregating  these  over  the 
total  spectrum  of  events  and  their  associated 
probabilities. 

Top-down  Risk  Analysis:  Determining  the 

critical  parameters  of  a  decision  andforting  the 
underlying  factors  among  alternative  choices  to 
become  visible  and  understandable. 

Joint  Approach:  Using  a  top-down  approach  to 
scope  a  problem  and  specific  bottom-up 
approaches  to  provide  quantification  where 
needed. 

In  order  to  illustrate  these  approaches,  a 
hazardous  waste  disposal  example-  is  provided. 
Two  different  cases  are  shown  because  they  each 


illustrate  the  methodologies,  as  well  as  represent 
actual  cases. 

Bottom-Up  Risk  Approach  For  A  Hazrdous 
Material  Site 

Consider  an  inactive  waste  disposal  site  owned 
by  a  coroporation,  which  is  suddenly  suspected  of 
containing  previously  unidentified  hazardous 
wastes  (as  defined  by  the  Resource  Conservation 
and  Recovery  Act  of  1976  (RCRA)).  The  case 
under  consideration  is  an  actual  case  which  has 
been  modified  for  use  as  an  illustration  here. 
Identifying  the  case  is  not  important,  nor  are  the 
conclusions  drawn  necessarily  valid  for  the 
particular  case  in  question. 

An  old,  uncontrolled  disposal  site  on  a  farm  in 
the  midwestern  United  States,  consisting  cf  an 
unlined  pit  in  which  an  estimated  150  drums  of 
waste  material  were  buried  around  1970,  was 
brought  to  the  attention  of  EPA.  EPA 
performed  a  two-week  field  study  .from  which 
they  determined  that  the  wastes  came  from  a 
company-owned  hexacliorophene  manufacturing 
process  and  which  contained  2,3, 7,8 
tetrachlorodi ben zo-p- dioxin  (TCDD)  as  well  as 
other  materials,  such  as  trichlorophenol  (TCP) 
and  ethyleneglycol.  Bore  holes  were  taken  to 
determine  leakage,  and  the  disposal  pit  was 
partially  excavated,  uncovering  13  drums  —some 
empty,  others  ranging  from  near  empty  to  full. 
Samples  confirmed  the  presence  of  TCDD  from 
29  mg/kg  (ppm)  to  100  mg/kg  of  materials  in  the 
barrels  and  soils.  Based  or  the  field 
investigation,  excavation,  and  results  of  the 
sampling,  the  EPA  further  concluded  that 
immediate  action  was  necessary  to  protect 
human  health  and  the  environment.  This  decision 
necessitated  the  development  cf  a  short-term 
response  program  to  minimize  and/or  prevent 
the  release  of  contaminants  from  the  site  until  a 
method  permanently  ameliorating  the  hazard 
could  be  implemented.  An  immediate  and 
temporary  measure  was  taken  by  the  EPA.  The 
disposal  trench  was  capped  with  an  impermeable 
membrane.  Surface  water  was  d  verted  from  the 
site.  At  this  point,  EPA  contracted  with  an 
independent  contractor  to  undertake  a  three- 
month  study  to  determine  how  to  deal  with  the 
problem.  This  study  will  be  used  to  illustrate  the 
process. 

The  objective  of  the  study  was  to  dean  up  the 
site  or,  at  least,  to  ensure  it  would  not  impose 
risks  to  the  public.  As  a  result  of  the  initial 
study,  it  became  evident  that  no  methods  were 
presently  available  for  final  treatment  or 
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disposal  which  met  even  minimim  Criteria. 
Several  methods  were  in  the  experimental  stage, 
but  several  years  would  be  needed  to 
demonstrate  their  capability.  As  a  resiit,  the 
following  conclusions  were  established: 

a.  There  is  no  method  for  final 
disposition  at  the  present  time; 

b.  Temporary  storage  is  required  intil 
suitahle  final  disposal  methods  are 
available.  . 

Four  general  alternatives  were  considered: 

1.  Leave  buried; 

2.  Leave  buried,  but  install  and 
maintain  a  ground  water  monitoring 
system; 

3.  Excavate  and  store  material  on-site 
in  a  newly  (to  be)  constructed 
temporary  repository, 

4.  Excavate  and  transport  drums  via 
truck  to  an  alternate  disposal  site 
(and  store). 

The  contractor  then  attempted  to  develop  the 
criteria  reqiiredfor  each  remaining  alternative. 
Not  unexpectedly,  the  risk  determination  caused 
the  most  significant  prohlem: 

'The  major  hazard  to  human 
health  due  to  the  wastes  at 
the  site  is  assumed  to  be  the 
toxicity  of  dioxin  (TCDD); 
for  simplicity,  only  this 
hazard  is  considered.  An 
"expos ire"  is  considered  to 
occur  whenever  a  person 
comes  directly  in  contact 
with  TCDD  in  high  enough 
concentrations  that  the  dose 
of  TCDD  to  his  body  exceeds 
an  assumed  safe  level,  which 
is  taken  to  be  1  part  per 
trillion  (ppt)  of  body  wei^it. 

The  level  of  effect,  that  is, 
severity  of  health 

impairment,  produced  in  the 
exposed  person  by  this  dose 
of  TCDD  cannot  easily  be 
predicted,  and  therefore,  the 
person  is  counted  as 
potentially  subject  to  some 
adverse  health  effect. 

Depending  on  the  actual 
magnitude  of  the  dose,  which 
in  turn  depends  on  time 
duration  of  the  contact  and 
other  pharmacological 

factors,  the  actual  level  of 
effect  suffered  may  range 
from  a  mild  and  probably 
reversible  case  of  chloracne 
to  cancer  of  the  liver. 

In  order  for  exposure  tc f 
TCDD  from  the  trench  at  the 


site  to  occur,  a  certain 
amount  of  TCDD  must 
escape  from  the  trench, 
spread  from  the  site  via  some 
physical  environmental 

pathway  and  ultimately  enter 
the  human  body  directly.  An 
effort  has  been  made  to 
systmatically  .  consider  all 
possible  pathways  and  to 
identify  those  exposire 
scenarios  which  are  most 
credible  for  four  alternative 
actions." 

At  this  point,  the  contractor  made  two 
assumptions  upon  which  his  analysis  was  based: 
(1)  since  human  risk  to  TCDD  cannot  be 
quantitatively  determined,  any  exposure  has 
negative  impact  and  that  impact  is  simply 
measurable  by  counting  the  number  of  people 
potentially  exposed  to  any  amount  of  TCDD 
from  the  site,  and,  (2)  the  probability  of 
exposure  scenarios  and  the  number  of  people 
potentially  exposed  could  be  estimated. 

Table  1  summarizes  the  scenarios  which  were 
considered,  the  estimated  probability  of 
occurrence  and  the  maximum  number  of  people 
exposed,  as  well  as  approximate  costs  cf  each 
alternative  over  five  years.  The  exposure 
estimates  are  worst-case  estimates.  Table  2 
summarizes  the  data,  and  shows  the  results  of 
the  expected  risk  computations  in  the  last  three 
columns,  i.0.,  the  maximum  number  of  people 
exposed  multiplied  by  the  probability  of 
occurrence  of  possible  events.  Alternative  3, 
the  most  expensive,  turns  out  to  have  the  lowest 
expected  risk,  but  only  by  a  very  small 
difference  at  the  third  significant  figure.  The 
rest  of  the  report  delineates  the  design  criteria 
necessary  for  a  facility  to  satisfy  this 
alternative.  It  is  obviously  not  the  cost-effective 
alternative;  but,  is  it  the  lowest  risk  alternative? 
Since  both  the  probability  and  consequence 
estimates  were  only  very  rough  estimates,  the 
results  of  this  approach  do  not  provide  very 
satisfactory  answers. 

The  difficulty  with  this  approach  is  that  all  the 
errors  aggregate  along  with  the  basic  data;  since 
they  are  large  and,  to  great  extent, 
multiplicative,  they  dominate  the  analysis.  This 
same  kind  of  prohlem  exists  in  all  bottom  up 
analyses,  especially  those  using  fault  and  event 
tree  approaches  to  probabilistic  risk  analysis. 

Top-Down  Analysis  For  A  Hazardous  Waste 
Decision 

One  cannot  do  a  total,  in-depth  top-down  or 
bottom-up  risk  analysis  for  the  same  problem  — 
at  least  in  the  cases  studied  thus  far.  If  a 
bottom-up  analysis  has  already  been  done,  the 
top-down  analysis  looks  like  second  guessing 
(after  all  the  data  has  been  gathered).  In  reality 
this  happens;  in  fact,  the  analysis  undertaken 
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TABLE  1 

ABBREVIATED  SUTTIARV’  OF  CREDIBLE  EXPOSURE  SCENARIOS 


Estimates  of 

Alternative  Action 

Probability  of 

Maximum  Number  of 

(Cost  Estimate) _ 

Exposure  Scenario 

Occurrence 

Pfopi  f  Fxporfti 

1. 

ter31 

A. 

Sinkhole  -  contents  of 

0.01 

1,446 

150  DRUMS  TO  GROUND  WATER 

POP. -4. 3  MiLES  OF  SITE 

B. 

No  SINKHOLE  -  GRADUAL  LEAK  TO 

119 

GROUND  WATER 

0.90 

POP.-1.2  MILES  OF  SITE 

2. 

Leave  blried 

A. 

Monitoring  systems  works  for 

0.98 

0 

Install  and  maintain 

BOTH  CASES  ABOVE 

MONITORING  SYSTEM  FOR 
GROUND  HATER  ($10,000) 

B. 

System  fails 

0.03 

379  -  SINKHOLE 

119  -  LEAK 

3. 

Excavate  and  store  on 

A. 

Ccf-MON  ACCIDENT 

0,20 

2-3  WORKERS' 

site  ($2,500,000) 

B, 

Tornado  strikes  during  operation 

3  x  10‘5 

50 

C. 

Leach  of  residuals  after  removal 

.95 

0 

D, 

Sinkhoie  effect  on  residuals 

0.04 

67 

E. 

Worker  contaminated  by  human 

0.G25 

40  WORKERS 

FA  I  LIRE 

120  OFF-SITE 

A. 

Excavate  and  Transport 

A. 

Same  as  a-e  above  for  alternative  3 

($1,500,000) 

B. 

Truck  accident  on  road 

3.5  x  10‘7 

1-2  WORKERS 

C. 

Truck  accident  at  facility 

3.5  x  10'7 

10  WORKERS 

TABLE  2 

SUT1ARY  OF  ESTIfiATED  RISKS 

MAXIfUl  AND  AVERAGE  IUEERS  OF  PEOPLE  EXPOSED  TD  DANGEROUS  CONCENTRATIONS  OF  TCDD 


Alternative  Remedial  Action 

OUR  INC  SHORT  TERM 

korkere  Public 

on  aita  off  aite 

OUR  INC  IONC  TERM 

Workera  Public 

on  aite  off  aita 

Total 

on  aite 

Total 

off  site 

Contained 

Total  exposures 

1.  laave  burled 

0 

1446  eax 

14.46  ava 

0 

119  eax 

107.10  eve 

0 

121.6  ave 

121.6  ave 

2.  Install  4  eeintetn  a  groundwater  0 
exxiltoring  eye tee 


)79  an 
0.1)  av« 


0  119  an  0  5). 7  eve  5). 7  eve 

5). 55  ave 


).  Cxcevele  4  atora  neterlel 
on  aita 


4)  an  170  asx 
*  20.6  ava  2)  i« 


0  67  a  ax 

2.7  ava  20  ava  27.7  ava  48.3  ava 


4.  Excavate  ♦  transport  drum  via  4)  an  180  an  0  67  aw 

truck  to  5ynt«  facility  in  21.9  ava  25  »vt  2.7  ava  21.0  ava  27.7  ava  48.7  ava 

Verona,  Mo. 


a  "Average"  la  the  aexixue  nrnbtr  aultlplled  by  th*  eatieeted  probability  of  occurrence  aea  labia  8*2 
b  T)enger*ie"  reana  high  enough  1o  laad  to  a  dnae  of  1  ppt  or  greater  in  the  average  Txjnen  body!  m  drinking  «*»r,  this 
threshold  concentration  la  0.0)5  ppb. 
c  "Short  term*  mar*  during  excavation  period.  qvproxiMtely  1  eonth. 

d  "long  lam"  aaana  greater  than  1  year  ( assume  no  other  future  action*  era  taken  whicta  lead  to  increased  worker  exposure*). 
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above  was  reanalyzed  by  an  informal  top-dawn 
analysis,  which  led  to  the  rejection  of  the  chosen 
alternative  and  the  adaption  of  a  totally 
different  strategy.  However,  this  woiid  not 
serve  to  introduce  the  top-down  approach.  In 
cases  where  a  top-down  approach  is  initially 
undertaken,  there  is  no  need  to  gather  all  the 
data  and  then  do  a  bottom-up  analysis.  Thus,  the 
case  presented  here  is  merely  to  introduce  and 
illustrate  the  top-down  approach,  not  to  compare 
the  two  methods. 

This  case  involves  a  large  chemical  company 
whicn  produces  hazardous  waste, 'and  which  must 
find  the  means  to  dispose  of  these  wastes  in 
order  to  keep  their  primary  production  processes 
in  operation.  Neither  the  Environmental 
Protection  Agency  (EPA),  nor  local  authorities 
with  jirisdiction  over  land  fills  and  land  farms, 
have  issued  new  commercial  hazardous  waste 
site  permits  for  several  years.  The  existing 
capacity,  as  it  is  used  up  by  the  large  number  of 
waste  disposers,  can  result  in  large  cost 
escalations  for  using  the  remaining  capacity. 
Moreover,  liability  for  faiiire  of  hazardous 
waste  facilities,  under  the  Resoirce 
Conservation  and  Recovery  Act  (RCRA),  make 


all  participants  in  a  site  liable  for  environmental 
and  health  impacts  for  disposal  system  failures, 
regardless  of  how  much  and  what  kind  of  waste  a 
particiiar  disposing  organization  committed  to 
the  disposal  site.  Other  details  will  be  brought 
out  in  the  discussion. 

Top-Down  Risk  Analysis  Procedure 

Table  3  lists  the  procedural  steps  necessary  to 
undertake  a  top-down  risk  analysis  and  Figire  1 
provides  a  diagrammic  view  of  this  process. 
Each  step  will  be  explained  in  reference  to  the 
above-mentioned  problem. 

Step  1;  Identify  a  Minimum  Set  of  Critical 
Variables  —  Initially,  five  critical  variables 
were  identified:  regulatory  climate  for 
obtaining  permits,  cost,  environmental 
damage  and  liability,  work  stoppage  and 
strike  potential,  and  transportation.  The 
last  two  were  found  to  be  of  lesser 
importance,  and  were  omitted  in  the 
minimization  process. 

Step  2:  Provide  Gross  Scales  for  these 
Variables  —  Scales  of  high,  medium  and 
low  were  used,  and  the  meaning  of 
each  classification  is  shown  in 
Appendix  1,  but  summarized  in  Table 
4. 


Step  3:  Generate  a  Set  of  Scenarios  from 
the  Combination  of  the  Intersections  of  the 
Variable  Conditions  —  There  are  three 
variables,  each  with  three  levels  of  value, 
leading  to  27  separate  scenarios. 


TABU  3 

TOP-DOWN  RISK  ANALYSIS 
PROCEDURAL  STEPS 

1.  Identify  a  minimum  set  of  critical  variables 

2.  Provide  gross  scales  for  these  variables 

3.  Generate  a  set  of  combination  scenarios  of  the  intersections  of  the  variable  conditions 

A.  Develop  a  set  of  alternative  strategies  for  solution  and  a  problem  structure 

5.  Identify  the  critical  decision  makers 

6.  Have  each  (or  group  of)  decision-maker  determine  his  choice  of  alternatives 

FOR  EACH  SCENARIO  OR  NEEDED  INFORMATION  TO  MAKE  A  CHOICE 

7.  Identify  scenarios  in  which  decision-makers: 

A.  All  agree  as  to  selection  of  alternatives 

B.  Have  irresolvable  conflicts 

C.  Require  further  information 

8.  Find  means  to  resolve  conflicts,  if  possible.  If  not,  stop 

9.  Specify  and  conduct  required  studies  to  obtain  required  information 


•  WHAT  INFORMATION 


•  LEVEL  OF  PRECISION  REQUIRED 

•  DECISION  POINT  -  IF  KNOWN  ' 

10.  Analyze  the  results. 
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Key  decision 

VARIABLES 

Alternative 

solutions 


Figure  1.  Diagratiic  view  of  top-dcw  risk  analysis  process. 


TABLE  4 

SUfMARY  OF  SCENARIOS 


SCENARIO  OWNERSHIP  TECHNOLOGY 


1.  A,a,i 

Company 

2.  A,a,ii 

Company 

3.  A.a.lil 

Company 

4.  B.a.i 

Company 

5.  B.a.11 

Company 

6.  B.a.iii 

Company 

7.  C.a.i 

Company 

8.  C.a.ii 

Company 

9.  C.a.iii 

Company 

ID.  A,c,1 

Company 

11.  A.c.ii 

Off-site 

12.  A.c.iii 

Off-site 

13.  A,b,1 

Company 

14.  A,b,1i 

Company 

15.  A.b.iii 

Company 

16.  B,b,i 

Company 

17.  B.b.ii 

Company 

IB.  8,b,i11 

Undecided 

19.  B,c,i 

Company 

20.  B.c.ii 

Off-site 

21.  B.c.lii 

Off-si te 

22.  C.b.i 

Company/Off-site 

23.  C.b.i i 

Off-site 

24.  C.b.iil 

Off-site 

25.  C.c.i 

26.  C.c.ii 

27.  C.c.iii 

HT 

UND  (Cost  vs  Liability  Study) 

LF 

HT 

UND  (Cost  vs.  Liability  Study) 

LF 

HT 

HT 

HT 

(Cost  vs.  Liability  Study) 

LF 

LF 

UND  (Cost  vs.  Liability  Study) 

LF 

LF 

UND  (Cost  vs.  Liability  Study) 

LF 

LF 

UND 

LF 

LF 

HT/LF  Difficult  to  get  permit 
LF 

LF  vs. 

Value  of  continuity 
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This  combinational  problem  requires  that  the 
number  af  assigned  variables  and  values  be  kept 
low.  Scenarios  were  developed  for  each 
combination,  for  which  three  were  .trivial  and 
eliminated.  The  scenarios  are  short  descriptions 
of  the  outcome,  and  only  two  are  reported  here 
to  provide  illustration  —  the  rest  are  not 
included  here  because  of  lack  of  space.  Note, 
that  at  this  step,  likelihoods  of  occurrence  have 
not  been  assigned  to  the  scenarios. 

Scenario  1;  ■  Optimistic  Regulatory 

Climate.  High  Cost  Escalation,  High 
Environmental  Impact  -  A',a,i 

The  regulatory  system  has  stabilized  and 
permits  are  obtained  with  reasonable 
effort.  However,  existing  capacity  for  The 
next  five  to  10  years  is  inadequate,  crusirg 
contractors  to  raise  disposal  costs  at  very 
hi$i  levels  of  escalation.  Moreover,  thr: 
management  of  these  facilities  has  be*c; 
less  than  adequate,  and  major  site  leaks 
can  be  expected  to  occur  at  one  or  more 
sites' involving  wastes  which  may  not  be 
the  company's,  but  which  are,  nevertheless, 
indi sti ngui shabl e.  Thus ,  the  com  pan  y  Co  ul  d 
share  in  liabilities  resulting  in  temporary 
and/or  permanent  closure.  The  fact  that 
permits  are  easy  to  get  that,  in  the  long 
run,  competition  will  make  adequate 
capacity  available  at  reasonable  costs. 
The  timing  of  such  availability  in  the 
proper  locations  ,s  a  key  issue  which  must 
be  compared  to  the  amortization  period  for 
any  company-owned  facility.  In  either 
case,  the  potential  for  high  environmental 
impact  by  land  fill  makes  the  high 
technology  operation  attractive  on  its  own. 

Critical  Factors: 

Years  to  high  capacity,  competitive 
contractors  availability  vs. 
amortization  period  of  high 
technology  investment. 

Cost  escalation  factor  and  estimate 
of  potential  liability  vs.  cost  of  high 
technology  alternative. 

Scenario  2: _ Optimistic  Regulatory 

Climate,  High  Cost  Escalation,  Medium 
Environmental  Impact  -  A,a,h 

This  scenario  is  similar  to  Scenario  1, 
except  that  the  environmental  impact  and 
attendant  liability  is  greatly  reduced.  The 
high  cost  escalation  in  the  short  run  makes 
contractor  ownership  unattractive  and  the 
company  can  obtain  permits  for  their  own 
land  fills. 

Critical  Factors: 

Increased  cost  of  hi^i  technology 
operation  vs.  land  fills  must  be 


compared  to  cost  escalation  factors 
and  potential  '  liability  of  medium 
environmental  impact. 

The  company  ownership  decision  is 
also  dependent  on  the  years  to  high 
capacity,  and  competitive 
contractors  availability  vs.  the 
amortization  period. 

Step  4:  Develop  a  Set  of  Alternative 
Strategies  for  Solution  and  a  Problem 
Structure  —  The  following  three 
alternative  strategies  have  been  proposed: 

1.  Hipti  Technology  Facility  -  HT 
On-site 

Objective:  Maximize  destruction  of 
waste 

Incinerator  and  other  processes 
Minimum  land  disposal 
High  investment 
Company  owned  and  operated 

2.  Off-Site  Disposal  (current)  -  LF  Off¬ 
site 

Land  intensive 
Off-site 

Land  farming  —  land  fill 
Low  investment 
Contracted 

3.  Company  Land-based  System  -  LF 
Company-owned 

Land  intensive 
On-site 

Company  operated  and  owned 
Incinerate  some  wastes 

The  prohlem  is  basically  a  sequence  of  two 
decisions  plus  some  contingency 
alternatives  in  the  face  of  uncertainty  in 
several  critical,  uncontrolled  states  of  the 
regulatory  environment. 

Decision  Level  One:  Use  contractor 
waste  facilities  or  develop  and 
operate  company-owned  facilities. 

Decision  Level  Two:  If  the  company 
develops  arid  operates  its  own 
facility,  shoiid  it  be  a  land  fill 
operation  or  a  high  technology 
operation? 

Contingency  Alternatives 

1.  Level  One:  Contractor  Facility 
—Shoiid  the  company  develop  a 
land  fill  site  for  use  as  a 
contingency  if  contractor 
operations  are  to  be 
inter  rrupteef? 

2.  Level  Two:  Company-owned 
Land  Fill  Operation  —  How 
many  such  sites  are  there,  and 
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where  should  they  be  located? 

3.  Level  Two:  Company-owned 
High  Technology  Operation  — 
What  kind  of  technology, 
capacity,  and  location  is 
required?  Shodd  remaining 
land  fill  needs  be  fulfilled  by 
the  company  on-site,  by  a 
contractor,  or  by  some 
combination  of  these? 


Environmental 

Department 


Legal  Department 


Environmental 
requirements 
and  relations 
with  Federal, 
state  and 
local  agencies 

Liability 
and  legal 
ramifications 


Risk  Analysis  T earn 


This  decision 
Figure  2.  It  . 
and  Level  2 


problem  is  illustrated  in 
.nould  be  noted  that  Level  1 
decisions  are  not  totally 


Team  responsible 
for  conducting 
the  analysis 
and  making 


rigut«  2  Tr*«  «cruccur«  of  th«  d«ci«lon  probita. 


Step  5:  Identify  the  Decision  Makers  — 
There  are  many  decision  makers,  but  the 
division  manager  will  make  the  final 
judgment.  The  array  of  decision-makers 
are  as  follows: 


Division  Manager  Final  decision 

responsibility 


Production  Management 


Need  for 
continuity 
of  capacity 


Engineering 
Managem  ent 


Design  of 
on-site  waste 
facilities 
and  hazardous 
waste  stream 
identification 


Step  6:  Have  each  (or  group  of)  decision 
maker  determine  his  choice  of  alternatives 
for  each  scenario  or  the  information 
required  in  order  to  make  a  choice.  The 
results  of  this  effort  are  shown  in  Table  4 
for  each  of  the  27  scenarios.  Each 
scenario  is  assigned  a  number  in  column  1, 
identified  bv  the  coded  combinations  as 
shown  in  column  2.  Column  3  shows  the 
ownership  decision,  column  4,  the 
technology  decision  or  information 
required  to  make  the  decision. 

Step  7:  Identify  Scenarios  in  which 

Decision-Makers:  (a)  ail  agree  as  to  the 
selection  of  alternatives;  (b)  have 
irresolvable  value  conflicts;  (c)  recjnre 
further  information.  In  this  case,  no 
conflicting  situations  were  found  which 
could  not  be  resolved;  however,  two  studies 
(a  cost  vs.  liability  study  and  permit 
difficulty  vs.  value  of  continuity  study) 
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were  identified  for  the  undecided  cases, 
UND,  for  the  technology  decision.  The  one 
undecided  case  for  the  ownership  decision 
is  only  for  landfills  and  is  a  lower  level 
decision. 

Step  8;  Find  Means  to  Resolve  Value 
Conflicts  —  There  were  no  value  conflicts 
which  could  not  be  resolved  in  order  to 
select  the  preferred  alternative  for  a  given 
scenario,  except  for  the  outcome  of  the 
two  studies  to  be  conducted.  Some 
unresolvable  conflicts  can  be  eliminated  if 
the  probablity  for  a  scenario,  where  a 
conflict  exists,  is  so  low  as  to  be  of  little 
concern.  The  probability  of  a  scenario 
depends  upon  the  joint  probability  of  each 
value  for  each  scale.  Approximate 
probability  classes  are  usually  adequate  for 
this  purpose  as  long  as  the  participants  in 
making  assignments  understand  and  agree 
on  the  meaning  of  such  assignments 
(precision  of  the  scale),  although  the 
choice  of  values,  may  differ  (accuracy  of 
the  measurement).  Classes  and  probability 
values  might  be  assigned  as  follows: 


Value 

Probability 

Probability 

Assignment 

Range 

Value 

Used 

High 

0.8-1.00 

0.9 

Moderate 

0.2-0. 8 

0.5 

Low 

0.01-0.2 

0.1 

Very  Low 

Less  than  0.1 

0.01 

The  joint  probability  of  the 

values  of 

the 

critical  parameters  are 

found 

by 

multiplying  the  three  values  together, 

one 

for  each 

parameter,  to 

obtain 

the 

probability  of  the  scenario.  Probabilities 
less  than  one  in  a  thousand,  with  respect  to 
the  highest  probability  scenario,  might  be 
considered  insignificant  (if  the 
consequences  are  catastrophic  even,  lower 
probabilities  should  be  considered).  Note: 
in  the  top-down  approach,  this  is  the  only 
method  in  which  probabilities  are  used, 
i.e.,  the  probability  of  a  scenario 
occurring. 

If  irresolvable  value  conflicts  survive  at 
this  stage,  no  further  technical  studies  will 
aid  in  resolving  the  conflict.  A  decision 
involving  political  power  is  the  only  way  to 
resolve  such  issues. 

Step  J9:  Specify  and  Conduct  Studies  to 
Obtain  Required  Information  —  Two 
studies,  described  in  Step  7,  must  be 
specified,  and  then  carried  out.  In  each 
case,  the  required  information  must  be 
identified,  as  well  as  the  level  of  precision 
of  the  required  information,  and  the 
decision  point,  if  it  is  known. 


Cost  vs.  Liability  Study  —  Since  the  high 
technology  option  involves  higher  capital 
and  operating  costs  than  the  lowest 
techndgy  options,  these  costs  must  be 
offset  by  a  significant  reduction  in 
liability.  It  was  determined  that 
engineering  estimates  for  capital  and 
operating  costs  only  had  to  be  precise  to 
one  significant  figure,  and  that  time 
discounting  of  money  flow  was  not 
required.  The  cost  estimates  for  contractor 
facilities  are  already  provided  in  the 
parameter  scales  (shown  in  Appendix  1). 
Using  similar  cases  and  trends,  the  legal 
department  made  estimates  of  possihle 
levels  of  liability  for  a  range  of  impacts 
for  each  alternative.  These  estimates 
were  only  necessary  to  an  order  of 
magnitude  of  one  significant  figure  for 
very  large  sums  of  money.  In  addition,  the 
legal  department  made  a  relative  risk 
estimate  of  the  likelihood  of  liability 
claims  from  each  alternative  to  about  one 
order  of  magnitude.  No  decision  points 
were  developed  for  this  study,  except  to 
express  the  scarcity  of  capital  funds. 

Difficulty  of  Getting  a  Permit  vs.  Value  of 
Continuity  Study  —  This  was  a  two-part 
study  aimed  at  how  long  the  production 
operation  could  run  if  no  land  fill  capacity 
was  available,  and  what  short  term  fixes 
could  be  undertaken,  such  as  interim 
permits  or  special  appeals.  These  studies 
were  strategic,  rather  than  tactical,  and 
were,  except  for  the  production  run  time 
to  shutdown,  qualitative  in  nature. 

Step  10:  Analyze  the  Results  —  The 
results  of  these  studies  are  then  added  to 
the  analysis.  The  format  for  the  results  of 
the  cost  vs.  liability  study  is  shown  in 
Table  5.  This  is  then  used  to  perforn  an 
indifference  analysis  as  shown  in  TaUe  6. 
This  indifference  analysis  shows  the 
probability  balance  point  where  the  actual 
estimite  of  probability  of  scenario 
occurrence  is  compared,  leading  to  a 
choice  of  one  scenario  over  the  other.  The 
indifference  probability  is  the  calculated 
decision  point  as  opposed  to  the  specified 
decision  point  (if  it  is  specified  a  prion). 

In  addition,  the  relative  likelihood  of  costs 
and  libilities  can  be  found,  as  shown  in 
Table  7.  Note  that  the  numeric  results  of 
the  analysis  are  not  given,  since  the  actual 
values  used  were  proprietary.  A  similar, 
but  abbreviated  analysis,  was  conducted 
for  the  second,  yet  qualitative  study. 

All  the  surviving  scenarios  are  then 
grouped  by  the  three  alternatives,  and  the 
sixn  of  the  probabilities  of  the  scenarios 
leading  to  each  alternative  is  determined. 
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This  is  the  probability  of  selecting  that 
alternative  and  having  made  the  right 
decision.  A  "what  if"  analysis,  as  shown  in 
Table  8,  can  then  be  used  to  determine  the 
cost  of  being  wrong.  The  decision  is  made 
by  maximizing  the  correctness  and 
minimizing  the  cost  of  being  wrong.  A 
dominant  solution  is  one  whose  probability 


of  being  correct  is  higher  and  whose  cos’ 
of  being  wrong  is  lower  than  another 
alternative,  which  is  dominant.  For  the 
case  involving  multiple  dominant 
strategies,  the  propensity  for  risk  of  the 
decision  maker  can  be  considered  in  term 
of  minimaxor  maximin  choices. 


TABuE  5 
ANALYSIS  1 

COSTS  AND  ENVIRONMENTAL  LIABILITY 


ALTERNATIVES 

COST 

PROJECTIONS 

SIZE  OF 
LIABILITY 

PROBABILITY 

OF  ENVIRONMENTAL 
DAMAGE 

High 

Mod 

Low 

Hign 

Mod 

Low 

Company  -  Land  Fill 

C,I 

CI2 

CI3 

hi 

L12 

L13 

PI 

Company  -  Land  Farm 

4l 

C22 

C23 

L21 

42 

43 

P2 

Company  -  Incinerator 

C31 

C32 

C33 

L31 

L32 

L33 

P3 

Contractor  -  Land  Fill 

C4I 

Ct2 

C43 

L«l 

42 

43 

P4 

Colt*  and  Liabilities  ere  0*er  Uie  Operating  Lile  ol  ttte  Facilities  -  Liabilities  Can  Eatend  Futher. 


TABLE  3 

STEP  1  Indifference  Analysis  1  (continued) 

P^  is  the  probability  of  damage  claims  of  a  contractor  landfill  over  its  life.  We  oo  not 

KNOW  THE  VALUE  OF  P^  EXACTLY.  We  00  HAVE  SOME  ESTIMATES  OF  RELATIVE  RISK  OF  THE  OTHER  OPTIONS 
IN  RELATION  TO  P(|,  I.E., 

pi  •  a1pA 

P2  -  AjPj 

Pj  -  a3p2 

P1  ■  Alp<i  *  a1ApA 
P2  -  a1a2p1)  -  A21|P1) 

p3  '  AlA2A3pP'  A3Api) 

The  inoifference  probability  is  founo  by  taking  any  two  options  ano  Ct j  ♦  PjL, j  *  CKL  +  pkL,j 

(Note:  p^  must  always  be  positive,  i.e.,  higher  cost  cases  must  have  lower  liability  expectation.) 
This  can  be  solved  for  all  combinations  of  Cost  and  Liabilities  to  fino  the  range  of  oecisioh 

VALUES  OF  P^  FOR  EACH  COMBINATION  OF  ALTERNATIVES,  I.E.,  NINE  CONDITIONS  FOR  EACH  COMPARISON  OF 
ALTERNATIVES. 


A ^  IS  THE  RELATIVE  IMPROVEMENT  IN  RELIABILITY  OUE 
TO  COMPANY  TOTAL  C'NTROL  OVER  A  UNO  FILL. 

A2  IS  THE  RELATIVE  IMPROVEMENT  IN  RELIABILITY  OF 
A  UNO  FARM  OVER  A  LAND  FILL. 

Aj  IS  THE  IMPROVEMENT  OF  AN  INCINERATOR  OVER  A  UNO  FARM. 
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TABLE  7 

STEP  2  Relative  Likelihood  of  Costs  and  Liabilities  Analysis  (continued) 

If  information  on  the  relative  likelihood  of  costs  are  available,  then  an  average  cost 

CALCULATION  COULD  BE  TRIED. 

Ci  ’  Bilcil  +  bi2ci2  +  bi3ci3  WHERE  Bil  +  Bi2  +  Bi3  =  1 

A  SECOND  METHOD  IS  TO  ORDER  THE  NINE  CALCULATIONS  IN  ORDER  OF  THEIR  RELATIVE  LIKELIHOOD 
WITH  A  CALCULATION  BESIDE  EACH  SHOWING  THE  RELATIVE  LIKELIHOOD. 

This  can  also  be  done  for  liabilities,  such  that 

Li  =  cilLil  +  ci2Li2  +  ci3Li3  where  cil  +  ci2  +  ci3  =  1 
Then,  when  ordered  by  relative  likelihood,  the  order  is  calculated  by  b j jCkl . 


TABLE  8 
ANALYSIS  3 
WHAT  IF?  ANALYSIS 

The  costs  or  opportunities  involved  in  selecting  an  alternative  i  based  upon  more 
favorable  conditions  for  alternative,  k  (see  Analysis  1). 


WICj k^jl  a  Cjj  "  CK  -  (Lkl  -  L,j)  +  Other  Costs,  if  any, 
There  are  no  probabilities  involved  since  the  scenario  happens. 


There  are  9  x  6  *  59  such  combinations.  But,  only  those  combinations  made  feasible 
by  a  change  in  regulatory  climate  or  siting  climate  should  be  investigated.  Moreover, 
AFTER  A  SPECIFIC  ALTERNATIVE  IS  CHOSEN,  THE  WHAT  IFS  REDUCE  TO  9  X  3  -  27, 


In  each  case,  the  studies  which  were  undertaken 
were  relatively  imprecise  estimates  made  by 
knowledgeable  personnel  with  a  minimum  of 
expended  resources  for  acquiring  data.  The 
subsequent  manipulation  of  the  data  was  to 
extract  the  maxim  no  available  information  from 
it.  The  indifference  analysis  provided  the 
decision  point  where  the  probability  estimates  of 
scenarios  could  be  ascertained.  Emerging 
dominant  strategies  force  a  decision.  In  other 
cases,  the  final  choice,  involving  the  propensity 
■for  risk  of  the  decision  maker,  is  presented  in 
decision-making  terms  which  decision  makers  at 
all  levels'can  understand. 

The  dominant  feature  of  the  approach  is  to 
focus,  very  early,  on  the  critcal  decision  factors 
and  only  obtain  bottom-up  type  data  for  the 
UND  conditions  to  a  limited  level  of  precision 
and  a  limited  data  gathering  effort.  The  use  and 
manipulation  of  data  for  maximum  information 
for  the  decision  is  stressed  rather  than  acquiring 
data  before  its  utility  in  the  decision  process  is 
ascertained.  Gathering  precise  data  is 
expensive,  manipulating  small  amounts  of 
imprecise  data  is  not. 

A  major  advantage  of  the  approach  is  that  it 
forces  the  decision  analyst  to  communicate  with 
the  decision  participants  in  mutually 
understandable  language.  This  is  accomplished 
by  developing  understandable,  mutually  exclusive 
and  collectively  exhaustive,  and  descriptive, 
scale  values,  outcome  scenarios,  alternatives, 
and  probability  assignments.  This  process  is 
built  in,  and  may  be  the  most  useful  aspect  of 
the  approach 

On  the  other  hand,  the  process  is  designed  for 
organizations  whose  various  facets  have  the 
same  objectives  yet  different  perspectives  on 
how  best  to  achieve  these  objectives.  The 
identification  of  conflicts  for  outcome  scenarios 
can  otherwise  bring  out  hidden  agendas.  For 
organization"  which  have  goal  conflicts  and 
intemedne  antagonisms,  this  process  might  be 
quite  upsetting. 


Use  of  the  Top  Down  Approach  in  the  Defense 
Department  Acquisition  Process 


Because  the  process  is  top>-down,  its  use  must  be 
made  in  the  same  manner,  i-e.,  starting  from  the 
top.  It  will  best  be  used  at  the  beginning  of  the 
acqusition  in  order  to  determine  performance, 
cost,  and  schedule  requirements  for  all  users, 
many  of  whom  may  have  conflicting 
requirements  (as  opposed  to  conflicting  agendas). 
Conflicting  requirements  can  be  evidenced  by 
developing  scenarios  for  different  outcomes  of 
performance  level,  schedule  and  cost 
combinations,  and  a  range  of  alternatives. 
Conflicting  agendas  may  become  visible,  and  this 
may  either  help  to.  provide  compromises  or 
exacerbate  the  situation. 


The  tof>-down  approach  promises  a  better 
rapport  between  the  users,  dedsion  analyst,  and 
dedsion  makers.  This  promise  can  only  be 
ascertained  by  attempting  to  use  it  on  several 
tests  or  real  cases. 


Footnotes 

*  "Technical  Study  and  Remedial  Action  for 
Denny  Farm  Site  1,  Aurora,  Missou-i"  (Final 
Report),  Document  No.:  EFSR80-09-Q105,  TDD: 
F7 -8006-01,  EPA  Contract  No.:  68-01-6056, 
Ecology  and  Environment,  Inc.,  September  15, 
1980,  p  B-l. 


APPENDIX  1 

MEANING  OF  VALUE  ASSIGNMENTS  TO 
CRITICAL  VARIABLES 

1.  THE  REGULATORY  CLIMATE  FOR 

ALLOWING  PERMITS  FOR  HAZARDOUS 
WASTE  SITES 

The  Environmental  Protection  Agency, 
under  RCRA,  is  responsible  for  permitting 
hazardous  waste  disposal  sites.  This,  in 
turn,  can  be  modified  by  state  and  local 
regulations  and  pressures.  Three  different 
scenarios  exist  for  alternate  regulatory 
conditions  in  the  future.  At  the  present 
time,  the  outlook  for  permits  is  very 
confusing  and  obtaining  permits  is  very 
difficult.  EPA  is  in  a  transition  status  and 
a  permitting  program  with  definitive 
criteria  does  not  yet  exist.  Obtaining  new 
permits  under  the  present  case  is  either 
difficult  or  non-existent. 


CASE  A:  Optimistic  Outlook 
EPA  comes  out,  in  the  very  near  future, 
with  reasonable  regulations  and  criteria  for 
obtaining  permits  for  hazardous  waste 
sites.  The  criteria  may  be  exacting  and 
sometimes  difficult  to  achieve,  but  the 


means  for  meeting  the  criteria  are 
unambiguous  and  straightforward  and, 
when  met,  permits  are  issued.  Permitting 
could  even  be  easier  to  obtain  than 


indicated  above,  i.e.,  the  dimate  could  be 


even  more  optimistic,  but  this  would  not 
have  much  affect  on  the  decision  criteria 


here. 


CASE  B:  Less  Pessimistic  Outlook 
EPA  regulations  on  permits  and  criteria  for 
obtaining  permits  are  issued,  but  the  means 
for  meeting  the  criteria  are  ambiguous, 
hard  to  predict,  lengthy,  and  unduly  subject 
to  public  and  other  pressures.  Permits  can 
be  obtained,  but  the  predictability  of  when, 
and  under  what  conditions,  permits  are 
issued  is  uncertain.  However,  the 
regulatory  dimate  Is  improved  over  the 
present  situation. 
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CASE  C:  Pessimistic  Outlook 
The  present  situation  continues.  Permits 
are  nearly  impossible  to  obtain  and  the 
confused  conditions  remain,  at  least,  for 
the  next  few  years. 

2.  COST  ESCALATION  OF  CONTRACTOR 
DISPOSAL  FEES 


Contractor  disposal  fees  are  bound  to 
escalate  when  there  is  inadequate  capacity 
for  disposal,  such  as  at  present.  Unless  the 
disposal  field  becomes  intensively 
competitive  for  wastes  of  the  type  the 
company  produces,  it  is  unlikely  that 
contractor  costs  for  disposal  will  be 
reduced. 

CASE  a;  High  Cost  Escalation 
Costs  of  disposal  escalate  at  rates  which 
are  increasing  fast  enough  to  offset  the 
investment  costs  in  the  high  technology 
operation.  At  the  present  status  of  this 
study,  the  investment  and  operating  costs 
have  not  yet  been  ascertained,  nor  are  the 
present  escalation  rates  greater  than  50% 
per  year.  However,  this  scenario  accepts 
that  upon  completion  of  cost  studies 
escalation  will  exceed  the  high  technology 
investment  costs,  making  such  investment 
feasible  on  an  economic  basis  alone. 

CASE  b:  Moderate  Cost  Escalation 
The  costs  escalate  at  a  rate  more  than 
enough  to  offset  the  costs  of  investment 
and  operation  for  a  company  land  fill 
operation  (cost  yet  to  be  determined),  but 
not  nearly  enough  to  offset  the  high 
technology  case. 

CASE  C:  Low  Cost  Escalation 

The  costs  escalate  at  or  near  the  inflation 

rate.  This  is  quite  different  than  present 

trends. 


3.  ENVIRONMENTAL  DAMAGE  AND  LIABILITY 


Whether  at  a  contractor  site  or  a  company 
site,  accidents  or  leaks  occur  requiring 
remedial  action,  fines,  and/or  liability 
claims. 


CASE  i:  High  Environmental  Impact 

Occurs 

A  major  off-site  leak  or  accident  occurs 
which  represents  a  serious  departure  from 
requirements.  Costly  remedial  actions, 
fines,  and  liability  for  the  off-site  public 
may  be  involved.  In  addition,  negative 
publicty  may  affect  corporate  image  and 
may  result  in  site  closure  or  restrictions. 

CASE  ii:  Moderate  Environmental  Impact 
Occurs 

Leaks  and  accidents  occ'jr  on-site  with 
minor  off-site  problems.  Remedial  actions 


may  be  necessary  and  fines  may  be  levied, 
but  liability  would  be  minimal.  Adverse 
publicity  and  pressures  on  site  operation 
might  occur. 

CASE  lii;  Negligible  Environmental  Impact 
Occurs 

The  site  performs  to  requirements  and  all 
accidents  and  leaks  are  retained  on-site, 
using  normal  procedures.  Problems  occur, 
but  are  within  the  normal  scope  of 
operations. 

4.  SITE  CLOSURE  SUSCEPTIBILITY 


This  variable  is  important,  but  only 
becomes  critical  after  the  first  three 
variables  have  been  determined.  There  are 
two  cases:  susceptible  and  less 

susceptible.  Some  of  the  reasons  for 
closure  are  regulatory  charges,  loss  of 
permit,  strikes,  public  pressure  for  closure, 
and  capacity  limit  reached.  For  each  of 
these  reasons  the  choices  of  Level  I  and  2 
decisions  become  more  or  less  susceptihle 
to  dosure.  Only  differences  among 
alternatives  need  be  considered. 

5.  TRANSPORTATION  OF  WASTES 


Transportation  of  wastes,  some  of  which 
may  be  hazardous,  becomes  a  problem  in 
only  two  respects:  0)  interruption  of 

transportation  by  state  or  local 
governments,  strikes,  etc.,  and  (2)  long 
distances  to  travel  which  provide  exposure 
for  accidents  and  higher  transportation 
costs.  The  basis  of  the  transportation 
derision  factors  is  whether  there  are  single 
or  multiple  waste  sites  and  their  locations. 
The  comparative  transportation  problem  is 
made  by  comparing  the  distances  and 
routes  among  the  technical  alternatives 
and  determining  the  relative  differences  in 
distance  and  localities  and  state  lines 
crossed.  The  risks  of  transport  per  mile 
are  low,  but  can  be  assessed  as  an 
additional  parameter  for  consideration  as 
will  the  costs. 
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ABSTRACT 

Risk  analysis  is  a  mathematical  field 
based  in  gambling,  investments  and  certain 
types  of  insurance  (life,  health  and 
accident.)  Attempts  to  adapt  analogous 
rationales  and  analytic  methods  to  public 
and  environmental  safety  issues  with  risky 
technological  projects  like  dams,  nuclear 
power  plants  and  marine  shipments  of 
hazardous  substances  have  sometimes  been  of 
dubious  value.  The  author,  based  on 
experience  in  maritime  safety/pollution  risk 
analysis,  has  recently  been  advocating  a 
shift  from  a  focus  on  risk  assessment  and 
risk  acceptance  analysis  towards  risk 
management.  The  purpose  of  this  paper  is  to 
adapt  that  same  reasoning  to  acquisition 
risk  analysis.  It  may  or  may  not  be  true 
that  some  acquisition  risk  analysis  studies 
„re  misdirected  at  non-existent  decisions  or 
at  decisions  that  do  not  depend  on  the 
degree  of  risk.  Similarly,  it  may  or  may 
not  be  true  that  some  acquisition  risk 
managers  fall  to  realize  their  true  and 
productive  role  In  the  risk  management 
process.  Finally,  it  is  possible  that  the 
Deming  management  philosophy  underlying  his 
quality  control  conepts  can  be  adapted  to 
aid  more  effective  DoD  systems  acquisition 
risk  management.  _ _ 

INTRODUCTION 

Risk  is  one  of  those  brief,  everyday 
words  which  Is  convenient  because  it  Is 
widely  and  conmonly  understood.  That  very 
convenience  Is  dangerous,  however,  because 
it  relies  on  imprecision. 

Risk  most  commonly  refers  to  personal 
danger  and  the  taking  of  chaces  with  personal 
safety.  There  Is  personal  risk  in  skiing, 
hang  gliding  and  race  driving  which  may  be 


compenstated  only  by  the  accompanying  thrill 
or  sense  of  accomplishment.  There  is  for 
many  an  exquisitely  heightened  perception  of 
risk,  with  little  real  risk,  In  riding  roller 
coasters.  Risk  frequently  includes  the 
connotation  of  an  unnecessary  danger,  as  in 
trying  to  jump  a  motorcycle  over  a  canyon, 
involving  an  apparent  chance  of  a  sudden  and 
spectacular  death.  That  association  leads  to 
a  common  popular  impression  that  risky 
vencures  are  unnecessary  and  careless. 

For  many  centuries,  maritime  commerce 
was  considered  risky  due  to  the  hazards  of 
the  sea,  storms  and  pirates.  The  economic 
ccncept  of  risk  most  probably  arose  In 
maritime  insurance,  wherein  dangers  to  the 
safety  of  ships,  goods,  and  crews  were 
compensated  by  money  from  a  fund  set  aside 
from  the  profits  of  several  risky  ventures. 

As  in  the  1600‘s,  today's  maritime  Insurance 
underwriters  do  not  rely  on  mathematical  or 
statistical  risk  analysis  to  decide  on  the 
acceptability  and  premiums  for  hull 
Insurance.  Instead,  they  consider  the 
specific  vessel,  Its  seaworthiness,  Its 
management,  Its  registry,  trade  and  crew. 

Other  types  of  Insurance  lend  themselves 
to  risk  analytic  (actuarial)  methods,  such 
as  life,  health,  auto  and  accident  Insur¬ 
ance.  In  such  cases,  there  are  very  large, 
fairly  homogeneous  populations  of  Insureds 
which  display  stable  patterns  of  loss 
experience  over  time. 

Risk  management  In  Insurance  generally 
means  diversification  of  risks  geographi¬ 
cally,  by  age  and  sex  of  Insured  and  by  type 
of  Insurance  to  minimize  the  chances  of 
catastrophic  loss  experience  In  a  single 
year.  For  large,  single  risks,  such  as  a 
chemical  ship  or  a  nuclear  power  plant,  the 
risks  are  managed  by  spreading  them  over 
numerous  underwriters. 
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Havltg  moved  from  a  concept  of  danger  to 
life  and  health  to  one  of  compensating  the 
loss  of  life  or  health  with  money.  It  Is  but 
c  small  step  to  view  risk  In  purely  economic 
terms.  In  gambling,  one  risks  some  amount 
of  money  the  hope  of  obtaining  more 
money,  but  with  a  generally  greater  chance 
of  losing  some  or  all  of  the  amount  ventured 
on  any  single  “play.“  This  context  also 
contributes  to  the  notion  that  risk-taking 
Is  foolhardy. 

Similarly  to  gambling,  one  can  apply  risk 
concepts  to  Investments  -  In' securities,  In 
new  plants  and  equipment,  or  in  new  sales 
territories  for  an  existing  product  line. 
These  contexts  have  led  to  specialized  forms 
of  analysis  such  as  portfolio  management, 
utility  analysis,  and  multi  attribute 
decision  analysis.  Such  tools  are  well 
suited  to  situations  where  there  Is  a  single 
decision-making  Individual  or  cohesive  group 
having  comnensurate  utility  functions,  where 
the  factors  bearing  on  the  decision  are  known 
and  can  be'  systematically  described,  and 
where  there  actually  is  a  definable  decision 
point  In  time  after  which  the  gamble  wins  or 
loses,  the  product  succeeds  or  falls  or  the 
Insured  lives  or  dies.  Systems  acquisition 
cost  risk  management  shares  some  of  the 
attributes  of  these  economic  contexts. 

Continuing  the  leaps  of  analogy,  a  person 
or  organization  may  take  a  professional, 
social  or  political  risk  by  associating  with 
any  activity  wherein  the  outcome  Is  not 
known  with  certainty.  Success  of  the 
venture  may  be  presumed  to  enhance  the 
reputation,  career,  credibility  and 
attractiveness  of  the  backer  whereas  failure 
will  have  the  opposite  consequences. 

To  complete  the  circle,  the  taking  of 
political  or  social  risks  Involves  dangers 
analogous  to  personal  safety  dangers  In  that 
failures  may  dramatically  affect  one's 
quality  of  life  and  one's  livelihood.  Even 
worse.  In  the  DoD  systems  acquisition 
environment,  a  backer  of  a  weapons  system 
procurement  may  later  have  his/her  life 
(along  with  many  others)  at  risk  In  battle, 
relying  on  the  practical  combat  effec¬ 
tiveness  of  the  system  procured. 

Having  explored  the  various  Informal 
connotations  of  risk,  let  us  now  turn  to 
risk  assessment  and  risk  management  Issues 
In  weapons  systems  acquisition. 

LIMITATIONS  OF  SINGLE  DECISION 
RISK  ASSESSMENTS 

In  the  context  of  safety /environmental 
risk  for  large  technological  projects,  It 
has  been  argued  that: 


Risk  acceptance  Is  a  myth. 

The  estimation  methods  used  are 
Inappropriate. 

The  risk  acceptance  concept  Is  In¬ 
appropriate  to  the  intended  decision  support. 

The  decision  context  often  does  not 
exist  at  all . 

"Risk  acceptance"  decisions  are  not 
even  theoretically  appropriate.  (1) 

There  is  no  question  that  appropriate 
environments  do  sometimes  exist  for  assess¬ 
ing  the  technological  and/or  cost  risk  of  a 
prospective  DoD  acquisition  to  determine 
whether  or  not  the  risk  is  acceptable. 
However,  the  appropriate  context  does  not 
always  exist.  The  following  questions  may 
be  useful  In  determining  whether  an 
acquisition  risk  assessment  Is  worthwhile. 

Is  there  a  decision  to  be  made?  If  the 
locus  (jurisdiction)  and  timing  are  well 
known  and  the  decision  options  have  not  been 
foreclosed,  then  some  form  of  decision 
analysis  may  well  be  appropriate.  There 
should,  however,  be  some  decisiveness  to  the 
decision.  Tentative,  easily  modifiable  or 
reversible  decisions  generally  do  not 
justify  as  extensive  or  solid  analytic 
support  as  more  definite  decisions. 

Is  risk  a  factor  in  the  decision?  Even 
If  a  definite  acquisition  decision  is  to  be 
made,  the  decision  may  not  be  greatly 
Influenced  by  any  conceivable  value  of  risk. 
The  decision  for  the  Apollo  space  program 
considered  technological  risk,  but  it  was 
not  a  serious  factor  In  the  decision.  If 
the  Soviets  could  get  a  Sputnik  Into  orbit 
In  1957,  by  golly,  we  could  surely  get  a  man 
onto  the  moon  before  1970.  The  decision 
issues  were  philosophical  and  political. 

What  set  of  values  would  prompt  us  to  want 
to  put  a  man  on  the  moon?  Was  it 
appropriate  to  spend  large  amounts  of  the 
national  treasure  on  a  venture  with  little 
assured  practical  value  to  humanity  when 
there  were  so  many  obvious  and  pressing 
human  needs  that  could  certainly  be  met  by 
alternative  allocations  of  funds  and  of 
scientific  and  engineering  brainpower? 

The  point  is  this:  not  all  risky 
ventures  are  decided  upon  on  the  basis  of 
their  risk.  If  no  conceivable  outcome  of  a 
risk  assessment  could  affect  the  decision, 
then  why  do  the  risk  assessment?  If  a  quick 
and  dirty,  even  a  qualitative,  risk  estimate 
is  all  that  is  needed,  why  do  an  elaborate 
quantitative  analysis? 


Systems  acquisitions  require  a  certain 
degree  of  service,  JCS/OSD  and  Congressional 
support  just  to  get  to  the  starting  line. 

It  can  be  argued  that  unless  some  minimal 
degree  of  support  Is  evident,  any  extensive 
risk  assessment  is  premature.  Conversely, 
if  the  political  support  Is  absolutely  rock 
solid,  that  may  be  because  of  considerations 
such  as  the  distribution  of  probable 
employment  by  Congressional  district  -  or 
some  other  factors  distinct  from  DoO  mission 
concerns.  In  such  cases,  technological  risk 
may  play  little  role  in  a  decision. 

Given  that  there  exists  a  decision 
context  and  that  “risk"  Is  In  some  sense  a 
factor  that  may  affect  the  decision.  It  does 
not  automatically  follow  that  risk  assessment 
will  be  helpful.  At  this  point,  we  have  to 
leave  the  layman's  loose  appreciation  of 
risk  to  consider  its  scientific  meanings. 
Historically,  risk  analysis  arose  out  of 
Insurance.  It  addresses  the  probability  of 
loss  and  the  magnitude  of  loss.  Within  the 
Dot)  systems  acquisition  risk  communi ty , 
however,  there  Is  not  a  widespread  adherence 
to  the  root  meaning  of  risk.  Some  analyses 
address  the  uncertainty  on  either  side  of 
expected  values  of  time,  cost  or  technical 
performance.  The  positive,  beneficial 
possibility  Is  not.  In  ar\y  classical  sense, 
a  risk.  Other  analyses  address  only  the 
probability  of  failing  to  meet  a  goal,  but 
do  not  address  the  consequences.  Other 
studies  address  only  the  range  of  conse¬ 
quences  that  may  occur,  but  not  the  likeli¬ 
hoods  associated  with  each.  Finally,  some 
of  the  studies  considering  a  range  of 
possibilities  address  only  the  mean  estimates 
or  expected  values  out  of  a  distribution. 

Rather  than  Insist  that  only  one 
definition  of  risk  can  be  used  (a  futile 
pursuit  Indeedi),  It  Is  more  pertinent  to 
note  that  whatever  Is  to  be  analyzed  In  a 
decision  risk  study  ought  to  be  relevant  to 
the  decision  to  be  made.  Some  decisions 
require  high  assurance  that  some  boundary  of 
cost,  time  or  performance  not  be  violated, 
but  are  then  quite  Insensitive  to  potential 
values  of  that  same  cost,  time  or  perfor¬ 
mance  measure  within  the' threshold. 

If  the  probabilities  associated  with 
various  values  of  cost,  time  or  performance 
are  to  be  analyzed,  several  Issues  should  be 
explored.  Is  there  an  appropriate  objective 
measure  of  the  criterion  of  Interest?  Must 
a  composite  measure  be  used  or  will  a  single 
measure  do  the  job  -  thrust-to-welght  ratio 
for  an  aircraft,  for  example,  rather  than 
simply  thrust?  Is  there  some  reason  to 
believe  that  a  specific  probability  distribu¬ 
tion  applies  or  should  alternative  candidate 
distributions  be  tested? 


If  the  potential  consequences  of  a 
decision  are  to  be  analyzed,  similar 
considerations  apply.  Is  the  measure  of 
consequence  appropriately  defined?  Is  a 
point,  interval  or  distributional  estimate 
desired?  Should  the  probability  and 
consequences  be  estimated  separately  with 
different  probability  distributions?  (In 
the  case  of  marine  oil  spills,  various 
researchers  have  determined  that  spill 
frequency  Is  best  represented  by  the 
negative  binomial  probability  distribution, 
but  that  the  size  of  spill,  given  a  spill, 
can  be  well  described  by  either  a  log-normal 
or  a  gamma  distribution.) 

Once  the  appropriate  type  and  measure  of 
risk  to  support  an  analysis  have  been 
determined.  It  Is  still  not  assured  that  a 
risk  assessment  will  be  of  any  value.  Some 
risks  are  dominated  by  their  uncertainties. 
There  are  cases  where  risks  are  uniformly 
distributed  or  where  the  boundary  values  of 
greatest  Interest  lie  well  within  a  single 
standard  deviation  of  the  mean.  In  such 
cases.  It  Is  not  at  all  clear  that  the 
Information  produced  by  a  risk  assessment 
will  be  of  any  use  In  the  decision. 

Sometimes  a  rough  estimate  can  show  this  to 
be  true,  avoiding  an  elaborate,  unproductive 
analysis. 

Finally,  let  us  assume  that  you  have 
thought  out  all  of  the  above  factors  and 
determined  that  a  decision  context  does 
exist,  that  the  decision  could  be  swayed  by 
a  sound  and  elaborate  risk  estimate,  that 
the  aproprlate  measures  and  distributions 
are  known,  and  that  the  risk  Is  not  likely 
to  be  dominated  by  Its  uncertainties.  Is  a 
decision  risk  assessment  now  known  to  be  a 
warranted  effort?  Not  quite. 

Even  the  best  justified  and  soundly 
prepared  risk  estimate  Is  subject  to  misuse. 
How  do  you  expect  the  risk  estimate  to  be 
used?  Can  It  be  presented  clearly  to  the 
decision-maker?  Will  Its  use  In  the  overall 
decision  analysis  be  structured  or 
unstructured?  Will  the  form  of  the  estimate 
support  the  structure  In  which  It  Is  to  be 
used?  How  much  of  the  Information  relevant 
to  the  decision  will  be  passed  along  to  each 
successive  level  of  command?  Regardless  of 
how  much  relevant  Information  Is  produced, 
how  much  will  actually  reach  the 
decision-makers?  If  caveats  are  vital  to 
the  decision  Input,  will  they  survive  the 
transmittal  process?  Such  questions  may  be 
viewed  as  beyond  the  purview  of  the  analyst 
or  the  staffer  who  orders  a  risk  assessment 
but  If  It  Is  totally  foreseeable  that  the 
valuable  components  of  a  risk  assessment 
product  will  not  enter  the  decision  process 
what  Is  the  justification  for  the  analysis? 
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One  advantage  of  writing  this  paper  after 
the  workshop  was  held  Is  that  It  Is  feasible 
to  refer  to  remarks  by  another  speaker.  The 
above  comments  at  the  workshop  were  grounded 
outside  the  DoD  systems  acquisition  process. 
Any  fear  that  they  might  not  be  relevant, 
however,  was  dispelled  by  Donald  Hurta,  who 
Indicated  that  a  great  many  analyses 
conducted  for  decision  support  fall  ever  to 
reach  a  decision-maker  or  to  be  used  In  a 
decision.  He  urged  us  to  consider,  as 
project  officers  buying  such  analyses  or  as 
performers  of  such  analyses,  why  we  continue 
to  be  parties  to  the  generation  of  unused 
studies.  There  are  many  answers,  of  course, 
but  there  are  no  comfortable  answers. 

SUCCESSFUL  ACQUISITION  RISK  MANAGEMENT 

Just  as  there  are  various  criteria  that 
must  be  met  for  a  risk  assessment  study  to 
be  useful  In  decision-making,  there  are  also 
criteria  which  can  help  to  determine  whether 
formal  risk  management  analysis  may  be 
useful. 

Above  all,  one  has  to  define  what  is 
wanted  In  order  to  manage  the  risk  of  not 
getting  what  you  want.  This  step  Is  perhaps 
the  hardest  of  all.  Does  a  systems 
^  acquisition  have  defined  requirements?  Of 
course,  there  Is  some  stated  operational 
requirement  In  the  file,  but  that  may  not  be 
relevant  to  risk  management.  A  requirement 
Is,  properly,  some  criterion  that  must  not 
be  redefined,  compromised,  violated  or 
Ignored.  If  a  criterion  were  truly  a 
requirement  of  a  procurement,  the  procure¬ 
ment  would  be  terminated  If  It  became 
apparent  that  the  criterion  would  not  be  met. 

A  design  goal,  unlike  a  requirement,  can 
be  compromised  or  violated  If  necessary.  If 
a  vehicle  Is  desired  to  have  twice  the 
payload,  range  and  reliability  of  Its 
predecessor  but  only  half  the  specific  fuel 
"v  consumption,  that  Is  unlikely  to  be  a  true 
requirement,  whatever  wording  Is  used  In  the 
documentation.  One  would  have  to  expect 
that  some  compromise  would  be  needed  at  some 
point  of  the  acquisition. 

Given  that  a  compromise  Is  likely,  are 
there  some  clear  Indications  of  acceptable 
contingency  positions?  Are  there  prlortles 
established  for  competing  criteria?  In 
short,  Is  there  sufficient  definition  of 
achievable  goals  to  make  It  plausible  that 
formal  analytic  support  would  aid  the  risk 
management  process? 

If  the  cost  risk  Is  to  be  managed,  will 
■  It  In  fact  be  feasible  to  “freeze"  the 

design  at  some  point?  Will  the  producer 
have  contractual  motivation  to  stay  within 
cost,  be  motivated  to  exceed  originally 


estimated  cost,  ur  be  motivated  to 
contribute  toward  the  sort  of  extensive 
redefinition  of  the  specification  that  would 
Justify  throwing  out  the  cost  celling? 

Given  that  there  is  some  definitive 
criterion  to  manage  to,  and  that  a  genuine 
risk  of  failing  to  meet  that  criterion 
exists.  Is  there  a  real  feasibility  of 
recognizing  and  accepting  the  failure  early 
enough  to  realize  a  substantial  savings?  In 
general,  does  the  managerial  discretion 
exist  within  the  procurement  system  to  allow 
the  risk  to  be  managed? 

Given  that  the  requirements,  goals, 
contingencies,  and  managerial  authority 
exist.  Is  there  a  defined  management  team 
which  Is  properly  motivated  to  monitor  and 
control  risk?  At  times  the  recognition  that 
a  gamble  (risk)  has  been  lost  or  that  some 
realignment  of  design  goals  Is  necessary  can 
be  painful  to  accept.  Historically,  bearers 
of  bad  news  have  occasionally  been  shot  by 
their  kings.  In  any  specific  acquisition, 
will  manager(s)  will  be  rewarded  for  taking 
the  ri ght  action,  regardless  of  what  right 
action  may  be  dictated  by  the  facts  of  the 
situation? 

Given  that  properly  motivated  risk 
managers  exist  and  are  defined,  do  they  have 
the  authority  and  resources  to  be  administra¬ 
tively  capable  of  sound  risk  management? 

Are  they  in  a  position  to  be  able  to  monitor 
the  risk  as  required  to  support  timely 
action?  Do  the  risk  managers  have  the 
necessary  training  and  analystlc  tools  to 
manage  risk? 

Given  that  a  capable  risk  manager  Is 
closely  monitoring  a  risky  acquisition,  are 
there  operational  definitions  of  risk 
actualization  triggers?  (Let's  paraphrase 
that.)  Risk  Is  a  potential  for  a  glitch. 

How  would  the  manager  recognize  the  glitch 
If  It  happened?  Are  appropriate  data 
gathered?  Does  the  management  team  know  how 
to  analyze  and  Interpret  them?  Are  suitable 
threshold  criteria  defined? 

Given  that  manager  detects  a  trigger 
signal,  can  he/she  select  and  Implement  the 
right  actions?  Are  the  actions  of  each  of 
the  other  parties  helpful  or  harmful  toward 
the  motivation  and  capabilities  of  the 
defined  risk  managers? 

DOERs  VERSUS  INFLUENCERs 

One  of  the  most  useful  recent  Insights 
into  (safety)  risk  management  Is  the 
recognition  that  risk  managers  can  be 
logically  divided  into  two  categories:  doers 
versus  Influencers. 
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Doers  have  been  defined  In  marine  safety 
analysis  as:  first,  the  ship  masters,  pilots 
and  watchstanders  who  have  direct  hands-on 
control  of  the  vessel  and,  secondly,  those 
ship  operating  company  executives  off-ship 
who  make  available  (or  fall  to  make 
available)  the  resources  and  policy  options 
required  in  the  hands-on  management  of  risk. 

Infl  uencers  are  everybody  el  se  whose 
actions  or  Inactions  affect  the  risk. 
Specifically,  in  the  marine  safety  case, 
there  are  the  government  authorities  who 
regulate  the  ships  and  people,  others  who 
provide  channel  maintenance  and  aids  to 
navigation,  insurers,  naval  architects  and 
ship  equipment  designers,  shipbuilders, 
equipment  manufacturers,  professional 
societies,  labor  unions,  schools,  and  anyone 
else  who  affects  the  capability  and 
motivation  of  the  doers  toward  risk 
management. 

The  key  point  is  that  only  the  doers  can 
directly  manage  risk.  Everybody  else  can 
only  influence  the  management  of  risk.  To 
the  extent  that  the  infl uencers  think  of 
themselves  as  direct  risk  managers,  they  are 
likely  to  be  ineffective  or,  quite  possibly, 
counterproductive.  To  be  effective,  they 
must  actively  consider  the  probable  impacts 
of  their  efforts  on  the  capability  and 
motivation  of  the  doers. 

To  illustrate  the  case,  bridge 
structural  engineers  have  recently  had  to 
devote  explicit  consideration  to  the  design 
of  bridges  so  as  to  minimize  the  risk  of  a 
ship  accidently  damaging  the  bridge.  Where 
such  accidents  cannot  be  absolutely 
prevented  (as  by  placing  the  supports  well 
clear  of  the  navigable  waters)  then  the  role 
of  the  bridge  designer  in  managing  this  risk 
is  to  maximize  the  capability  and  motivation 
of  the  ship  master  or  pilot  to  manage  the 
risk.  This  concept  is  as  vital  as  it  is 
revolutionary. 

A  parallel  has  to  exist  in  systems 
acquisition  risk  management,  but  it  is  not 
easy  to  draw  the  analogy  in  a  workable 
manner.  It  may  be  well  to  question, 
however,  whether  after  contract  award  the 
doers  may  not  be  limited  to  the  prime  and 
subcontractor  program  managers.  If  this  is 
true,  it  has  important  implications  for  the 
government  program  managers.  If  DoD  program 
managers  are  the  doers,  they  should  seek  to 
manage  the  risk  directly.  If  they  are 
infl  uencers,  then  it  becomes  important  that 
they  not  seek  to  manage  the  risks  directly; 
they  are  likely  to  foul  up  the  efforts  of 
the  actual  (contractor)  doers.  Instead, 
they  must  act  consistently  to  maximize  both 
the  motivation  and  the  capability  of  the 
doers  to  eanage  risk.  Most  of  that 


motivation  and  capability  has  :o  be 
established  in  the  contract,  including  the 
risk  management  resources  provided  for  in 
the  contract. 

Before  award,  there  may  be  a  succession 
of  doers  at  various  stages,  most  likely 
culminating  in  the  assigned  DoD  program 
manager  for  some  period  prior  to  actual 
award.  Even  at  that  stage,  however,  it  will 
be  extremely  important  for  the  DoD  program 
manager  to  so  structure  the  contract  and  to 
so  influence  selecting  the  contractor  that 
the  doer  functions  will  be  smoothly  and 
effectively  transferred  to  tne  contractor 
after  award. 

If  this  analogy  has  merit,  it  suggests 
that  the  following  are  all  Infl  uencers  and 
should  carefully  rethink  their  roles  in  that 
light: 

-DSARCs 

-DoD  Program  Managers 

-Congress 

-Writers  of  Military  Standards  and 
Military  Specifications 

-Professional  Societies 

-Schools  that  train  the  Doers 


DEMING's  14  POINTS 

W.  Edwards  Deming  is  famous  for  the 
impact  of  his  approach  to  the  use  of  quality 
control  statistics  on  the  subsequent  growth 
of  Japanese  productivity  and  competitiveness 
after  his  methods  were  introduced.  Dr. 
Deming  does  not  attribute  the  success  to  the 
statistical  methods  themselves  (nor  to 
quality  circles  and  other  phenomena  which 
become  involved  in  an  applied  quality 
management  program.)  He  views  the  success 
of  his  program  as  being  due  to  some 
fundamental  realizations  and  to  certain 
managerial  practices.  (2)  The  current 
author  is  attempting  to  develop  extensions 
of  the  Deming  approach  outside  the  basic 
context.  Paratheses  are  used  in  this 
section  to  distinguish  those  thoughts  from 
the  better  established  principles  of 
Deming's  work. 

The  fundamental  realization  must  be  tha 
quality  and  productivity  are  not  in  conflic 
but  in  harmony,  in  a  properly  managed 
enterprise.  The  inevitable  consequence  of 
lack  of  quality  is  waste,  and  waste  is 
counterproductive. 
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{It  seems  logical,  then,  that  Deming's 
concept  could  be  extended  into  safety  risk 
management.  By  analogy,  safety  and 
productivity  should  not  be  In  conflict. 
Accidents  waste  time  and  resources  and,  if 
they  Impact  quality,  that  impact  is  adverse. 
Thus,  lack  of  safety  is  counterproductive. 
The  idea  that  efforts  devoted  to  safety 
detract  from  productivity  must  be  due  to 
faulty  management.) 

(Having  al ready  extended  Deming's  basic 
principle  somewhat  tenuously,  can  we  stretch 
the  analogy  even  further  to  acquisition  risk 
management?  In  the  grand  view,  it  seems 
inevitable.  Over  a  large  number  of  systems 
acquisitions,  the  better  acquisition  risk  is 
controlled,  the  more  productive  in  terms  of 
DoD  mission  capability  will  be  the  systems 
acquired.  Presumably,  this  is  consistent 
with  having  higher  quality  weapons  systems 
than  under  conditions  wherein  acquisition 
risk  is  less  well  managed.) 

(It  is  less  apparent  how  to  adapt  the 
Deming  principles  to  the  prevention  of 
specific  accidents  or  to  the  management  of 
risks  in  a  specific  systems  acquisition.) 

The  reference  context  in  which  Deming  has 
worked  has  been  a  repetitive  industrial 
production  environment  where  parts  and 
materials  purchased  from  other  companies  are 
further  processed,  packed,  shipped  and 
marketed.  He  has  readily  extended  the 
applications  to  bank  teller  operations  and 
to  hospitals.) 

(Some  of  the  ways  the  reader  is  urged  to 
consider  Deming's  philosophy  is  for 
adaptation  to  generic  guidelines  and  control 
systems  for  DoD  systems  acquisitions  or  as  a 
contract  requirement,  somewhat  different 
from  current  quality  control  regulations,  to 
be  enforced  upon  production  lines  of  Defense 
contractors. ) 

What  distinguishes  Deming's  approach 
from  other  theorists  and  applications 
authorities  in  quality  control  is  the  14 
point  management  schema  which  Deming  Insists 
Is  necessary  to  create  the  conditions  under 
which  a  quality  control  program  can  flourish 
and  be  consistent  with  high  productivity. 
They  are  summarized  briefly  here;  the  reader 
Is  encouraged  to  read  reference  2. 

1.  Long  range  goals  are  more 
important  than  short  range  goals.  (Ke <p 
your  eye  on  what  you  are  really  after  - 
effective,  affordable  national  security.) 

2.  Managements  must  become  less 
tolerant  of  waste  and  "business  as  usual." 
(Never  mind  that  Defense  contracts  may  • 
display  a  stubborn  tendency  to  1091  cost 
growth;  you  will  not  allow  it. 4 


3.-4.  Demand  and  enforce  statis¬ 
tical  evidence  of  process  quality  control  in 
incoming  materials;  this  will  reduce  the 
number  of  contractors  willing  to  deal  with 
you,  which  is  good. 

5.  Constantly  Improve  the  system  of 
production  and  service;  you  cannot  do  this 
except  through  statistical  monitoring. 

6.  Modernize  training.  Use 
statistical  quality  control  methods  to 
monitor  training  effectiveness. 

7.  Improve  supervision,  mainly  by 
letting  the  supervisor  inform  upper 
management  concerning  conditions  that  need 
correction.  Do  not  allow  supervision  to 
mean  forcing  the  staff  to  work  with  whatever 
is  available,  regardless  of  how  poorly  it  is 
suited  to  the  work  or  how  poorly  it  is 
maintained. 

8.  Drive  out  fear.  Especially,  let 
people  ask  questions  without  fear  of 
reprisal  or  humiliation.  Let  managers  not 
fear  to  ask  workers  what  the  job  Is  and  how 
to  tell  whether  it  is  being  done  correctly 
or  not.  Let  the  workers  not  fear  to  ask 
more  than  three  or  four  times  what  work  is 
acceptable  and  what  is  not.  Let  a  crew  not 
fear  to  shut  down  a  production  line  for 
crucial  repairs  before  a  catastrophic 
failure. 

9.  Break  down  barriers  to  communication 
between  staff  areas:  operations  and 
maintenance,  engineers  and  technicians, 
plant  personnel  and  logistical/clerical 
staff,  research,  design,  recruiting, 
training,  public  relations. 

10.  Eliminate  numerical  goals.  Slogans 
such  as  “Zero  Defects",  "Improve 
productivity",  "100  days  without  a  lost  time 
accident",  are  motherhood  goals.  Nobody 
could  disagree  with  them.  The  problem  is 
that  they  do  not  enhance  the  capability  or 
the  motivation  of  anyone  to  do  a  better 
job.  They  advertise  Instead  that  management 
does  not  know  how  to  achieve  desirable  goals 
so  it  is  depending  on  the  work  force  to  find 
some  way  to  do  it. 

11.  Eliminate  counterproductive  work 
standards  and  quotas.  For  various  reasons, 
it  is  usually  undesirable  for  a  workforce  to 
exceed  a  quota  or  even  to  meet  it  with 
ease.  For  that  reason,  the  quota  becomes  an 
upper  limit.  If  management  and  workers  are 
cooperating  to  make  the  system  work  at  its 
best,  why  Impose  an  upper  limit  on  the 
Improvement? 
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12.  Allow  the  staff  pride  of  workmanship 
by  defining  quality  work  measurably  and 
meaningfully  and  appraising  It  consistent¬ 
ly.  People  who  are  rated  Inconsistently  for 
the  same  quality  of  output  will  retain 
neither  their  quality  standards  nor  their 
pride.  Because  systems  and  people  are 
variable,  output  Is  variable.  Quality, 
output  and  productivity  can  only  be 
meaningful  If  they  are  defined  statistically 
and  evaluated  by  consistent  statistical 
methods. 

13:  Institute  a  vigorous  program  of 
education  and  training.  Everybody's  job  Is 
changing  over  time.  When  statistical 
quality  (and  risk)  control  methods  are 
Instituted,  that  process  alone  changes  all 
managerial  and  engineering  jobs.  Among 
other  training,  provide  all  of  these  people 
training  In  at  least  the  rudimentary 
statistical  methods  needed  for  monitoring 
and  control . 

14.  Institutionalize  the  new  philosophy 
so  that  It  gets  a  continual  push.  Embedding 
this  concept  of  quality  ,id  risk  control  Is 
not  a  one-time  effort.  Every  one  of  these 
principles  will  run  out  of  steam  unless  the 
conmltment  to  them  Is  renewed. 

THE  BOTTOM  LINE 

Risk  management  should  mean  a  continuing 
commitment  to  the  requirements,  design 
goals,  costs  and  schedules  In  a  project. 

That  cannot  be  directly  accomplished  by 
anyone  other  than  the  direct  operating 
managers  at  any  acquisition  stage.  Others 
can  only  Influence  their  capability  and 
motivation  toward  risk  control.  Risk 
management  must  be  made  consistent  with  high 
quality  and  high  productivity.  Achieving 
any  or  all  of  these  requires  that  certain 
management  principles  be  In  force.  Demlng 
has  suggested  a  set.  It  Is  up  to  us  to 
adapt  or  replace  his  set  toward  defining 
workable  principles  for  systems  acquisition 
risk  management. 
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Y  ABSTRACT 

This  paper  presents  a  risk  management 
model  that  is  designed  to  identify,  assess, 
and  manage  risk  for  Defense  weapon  system 
acquisitions.  The  model  provides  a  detail¬ 
ed,  disciplined  process  that  establishes 
baseline  definitions,  principles  for  risk 
management,  and  diagrams  which  support  the 
risk  management  process. 

BACKGROUND 

Risk  management  for  the  DoO  system 
acquisition  process  is  not  an  exact 
science.  There  is  no  common  or  standard  DoO 
sanctioned  practice,  technique,  or  procedure 
for  identifying,  assessing  and  managing 
risk.  Furthermore,  there  is  no  universally 
accepted  definition  of  risk.  The  most 
common  form  of  risk  management  is  to 
quantify  the  probability  of  loss  in  dollar 
values,  much  like  the  Insurance  Industry. 
Dollar  values,  or  costs,  are  only  one-third 
of  the  risk  problem  for  DoD  Project 
managers.  The  other  two  factors  are  system 
performance  (quality)  and  schedule  (time) 
for  producing  the  system.  Therefore,  a 
valid  requirement  exists  for  a  disciplined 
process  that  will  facilitate  the  uniform 
treatment  of  risk  variables  and  allow  under¬ 
standing  of  risk  as  it  applies  to  the  DoD 
system  acquisition  process. 

PURPOSE 

This  paper  provides  a  risk  management 
model  that  can  be  used  to  Identify,  assess, 
and  manage  the  risk  of  the  three  areas  of 
the  system  acquisition  process  --  cost, 
schedule,  and  performance.  This  risk 
management  model  was  specifically  tailored 
for  the  full  scale  engineering  development 
of  a  major  surveillance  system. 

DEFINITIONS 

The  Introduction  of  terms  that  differ 
from  generally  accepted  definitions  Is 
neco'sary  to  establish  the  baseline  for  the 
system  acquisition  risk  management  model. 
These  terms  "are  defined  below. 


a.  Risk  -  The  resultant  product  of  the 
probability  of  failure  and  the  consequence  of 
that  failure  for  any  preset  goal  (expressed 
in  terms  of  performance,  cost,  and/or 
schedule  criteria). 

b.  Probabll ity  of  Failure  -  The 
assessed  “"value  of  BEe  likelihood  of  not 
meeting  the  goal  or  criterion  for  an 
activity.  The  probability  of  failure  is  best 
expressed  numerically,  but  may  be  expressed 
in  qualitative  terms.  Figure  1  may  be  used 
to  convert  qualitative  terms  to  numeric 
values . 

c.  Uncertainty  -  Insufficient  knowledge 
(information)  to  assess  risk  in  concrete 
terms.  (Uncertainty  Is  often  confused  with 
risk  and  used  incorrectly  as  a  synonym  for 
risk.) 

d.  Risk  Assessment  -  A  comprehensive 
and  disciplined  process  for  assigning  factors 
of  risk  (probability  and  consequence  of 
failure)  to  particular  activities;  also  the 
product  of  the  foregoing  process. 

e.  Risk  Management  -  Developing  and 
Implementing  alternate  courses  of  action  to 
reduce  factors  of  risk. 

f.  Risk  Analysis  -  A  procedure  for  the 
division  of  a  project  into  activities  and  the 
uniform  examination  of  these  elements  to 
determine  the  probability  and  consequence  of 
failure;  also  the  product  of  such  a  process. 

g.  Failure  Identification  -  The  process 
of  determining  areas  o?  probable  failure 
within  an  activity  for  subsequent  analysis, 
assessment,  and  management. 

h.  Consequence  of  Failure  -  The 
quantlflaFTi  loss  of  money,  time  or  quality 
resulting  from  the  failure  to  meet  a  goal  or 
criterion. 

1.  Risk  Threshold  -  The  upper  limit  of 
risk  thal  wi  11  Be  accepted  by  the  project 
manager  for  any  activity  or  element  of  a 
project. 
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Order  of 

Adjectival 

Chances 

Probability/ 

Likelihood 

Qualitative  Terms 

Rating 

in  10 

Percent 

Near 

Virtually  (almost)  certain 

Very  High 

9 

.99/99 

Certainty 

We  are  convinced, 

of 

Highly  probably 

Failure 

Highly  likely 

1 

Probably 

Likely 

High 

8 

Will 

We  believe 

Fail 

We  estimate 

Chances  are  good 

7 

It  is  probable  that 

6 

Even  Chance 

Chances  are  slightly  better  than  even 

Medium 

5 

of  Success 

Chances  are  about  even 

or  Failure 

Chances  are  slightly  less  than  even 

4 

Improbable 

Probably  not 

Low 

3 

Failure 

Unlikely 

We  believe  .  .  .  not 

2 

.10/10 

Near 

Almost  impossible 

Very  Low 

1 

.01/1 

Impossibility 

Only  a  slight  chance 

of  Failure 

Highly  doubtful 

Figure  1.  ESTIMATIVE  TERMS,  DEGREES  OF  PROBABILITY,  AND  PERCENTAGES 


PRINCIPLES  OF  RISK  MANAGEMENT 

Risk  management  entails  a  disciplined 
process  that  operates  within  the  scope  of 
four  major  principles. 

a.  Risk  is  present  in  the  system 

acquisition  process  only  when  there  are 
established  goals,  objectives,  or  criteria. 

b.  Risk  must  be  assessed  on  elements, 

or  activities,  of  the  total  system 

acquisition  for  management  of  the  parts. 

c.  The  risks  assigned  to  elements,  or 

activities,  cannot  be  sunmed  and  averaged 
for  risk  to  the  total  system  acquisition. 

d.  Uncertainty  limits  the  ability  to 
assess  risk  (l.e.,  probability  of  failure 
and  consequence),  but  does  not  increase  the 
risk. 


RISK  MANAGEMENT  MODEL 

The  risk  management  model  addresses  more 
.than  the  elements  of  risk,  and  works  in  con¬ 
cert  with  corononly  accepted  project  manage¬ 
ment  practices,  procedures  and  techniques. 
The  model  is  designed  to  fit  within  the 
current  management  structure  as  a  complemen¬ 
tary  tool  rather  than  replacement  for  or 
addition  to  existing  techniques.  The  model 
uses  the  outputs  from  the  work  breakdown 
structure  and  networks  such  as  PERT,  CPM  or 
PCM. 

Figure  2  depicts  the  overall  diagram  of 
the  risk  management  model.  This  Is  a  com¬ 
prehensive  overview  of  the  work  required  to 
establish  the  risk  Identification,  assess¬ 
ment,  tradeoff  activities,  and  finally,  the 
end  ranagement  of  risk  during  project 
implementation.  The  flow  of  activity  is 
shown  by  arrows  and  lines  in  &  logical 


Figure  2.  RISK  ASSESSMENT  DIAGRAM 
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sequence  from  top  to  bottom.  The  activities 
are  specifically  ordered  from  top  to  bottom 
as  a  natural  sequence  for  addressing  each 
one  and  to  limit  biases  by  establishing 
thresholds  prior  to  the  estimating  process. 
The  procedures  for  using  this  model  are  as 
follows: 


Project  Work.  Identify  all  the  work  to 
be  accompl ished  from  start  to  finish  (with 
the  knowledge  that  subsequent  changes  will 
normally  be  required).  The  totality  of  the 
identified  work,  even  in  large  packages, 
will  provide  an  upper  bound  for  resources 
required. 


Project  Goal.  The  goal  of  the  system 
acquisition  is  often  stated  in  the  Project 
Office  charter  or  another  document  that  des¬ 
cribes  the  authority  and  responsibility  of 
the  project  manager.  If  there  is  no  stated 
goal,  one  must  be  developed  prior  to  pro¬ 
ceeding.  The  goal  provides  the  focus  for 
development  of  subsequent  work. 


Risk  Threshold.  Before  proceeding 
further,  establish  the  degree  of  risk  that 
is  acceptable  for  any  one  activity  in  cost 
(money),  schedule  (time),  or  performance 
(quality).  This  designation  of  the  Risk 
Threshold  (RT)  is  a  predetermined  criterion 
from  v*iich  all  activities  will  be  measured 
(see  Figure  3).  It  provides  a  uniform,  un¬ 
biased  standard  that  will  cause  acceptance, 
rejection,  or  reassessment  of  activities. 
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Consequence  of 

Failure 

Probability  of  Failure 

_ 

Very 

High 

High 

Medium 

Low 

Very 

Low 

Very  Severe 

.9 

.8 

.7 

.6 

.5 

Severe 

.8 

.7 

.6 

.5 

.4 

Moderate 

.7 

.6 

.5 

.4 

.3 

Little 

.6 

.5 

.4 

.3 

.2 

Very  Little 

.5 

.4 

.3 

.2 

.1 

Figure  3.  RISK  QUANTIFICATION  MATRIX 


NOTES:  (1)  The  decision  to  accept  or  reject  any  risk  is  the  function  of  the 

project  manager.  The  decision  may,  for  example,  be  made  to  accept 
any  risk  of  0.4  or  less  without  tradeoff  studies,  and  to  conduct 
tradeoffs  on  all  risks  greater  than  0.4.  If  tradeoffs  cannot  reduce 
the  risk  to  0.4,  then  the  project  would  be  deferred  pending  approval 
of  the  sponsor. 

(2)  To  determine  value  (measure  of  success)  for  any  activity,  use 
the  formula  of  1  -  risk.  For  example,  a  very  high  probability  of 
fanure  intersecting  with  a  severe  consequence  is:  1  -  0.8  ■  0.2. 
Thus,  the  value  of  this  activity  is  0.2. 

(3)  The  use  of  colors  may  assist  in  identifying  the  zones  of 
acceptable,  marginally  acceptable,  and  unacceptable  risk.  Red  may  be 
used  for  0.7  through  0.9,  yellow  for  0.4  through  0.6,  and  green  for 
0.1  through  0.3. 
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Work  Packages.  The  total  work  is  di¬ 
vided  into  the  work  packages  that  represent 
well  defined  activities  of  a  work  breakdown 
structure  for  managerial  purposes.  MIL-STD- 
381A  should  be  used  as  a  guide  for  standard¬ 
ization  and  to  ensure  all  support  items  are 
Included. 

Network.  Convert  the  work  breakdown 
structure  to  an  interdependent  network  to 
place  activities  in  the  proper  sequential 
interrelationship.  '  This  network,  in 
addition  to  establishing  interrelationships, 
facilitates  the  computation  of  the  total 
time  required  to  complete  the  project. 
Another  advantage  of  the  network  is  the 
capability  to  predict  the  start  and  finish 
of  individual  activities. 

Allocation  of  Performance,  Time,  and 
CostT  Allocate  each  activity  ’n  the  network 
a  performance  criterion,  a  time  for  comple¬ 
tion,  and  a  cost  value.  These  baseline 
figures  for  each  activity  are  the  values 
which  will  be  used  to  assess  the  prob¬ 
abilities  of  achieving  the  individual 
values.  Record  the  performance,  time,  and 
cost  values.  A  format  similar  to  Figure  4 
may  be  used  for  this  purpose. 

Probability  of  Failure  for  Activities. 
The  values  previously  assigned  for  cost, 


time,  and  performance  are  evaluated  using 
Figure  5  and  a  probability  value  (number 
between  0.0  and  1.0)  assigned  for  each  area 
and  each  activity.  The  estimative  terms  of 
Figure  1  may  be  helpful  in  making  the  evalua¬ 
tion.  This  evaluation  and  assignment  of 
probabilities  should  not  be  accomplished  by 
the  same  individual  who  made  the  allocation 
of  values.  Record  the  probability  of  failure 
data.  (See  Figure  4.) 

Consequences  of  Failure  for  Activities. 
Fol lowing  the  assessment  for  the  probability 
of  failure,  a  consequence  of  failure  must  be 
derived  for  each  activity.  The  criterion  for 
consequence  of  failure  is  outlined  in  Figure 
6.  Record  the  consequence  of  failure  data 
(See  Figure  4.) 

Risk  Threshold.  Using  Figure  3,  deter¬ 
mine  tFTe  risk  value  for  performance,  time, 
and  cost  in  each  activity  by  determining  the 
intersection  of  the  probability  and  conse¬ 
quence  of  failure.  The  resultant  value  can 
be  compared  to  the  Risk  Threshold  established 
previously.  Each  performance,  time,  and  cost 
for  each  activity  that  is  equal  to  or  less 
than  the  Risk  Threshold  can  be  temporarily 
set  aside.  Areas  of  activities  that  exceed 
the  Risk  Threshold  must  be  evaluated  for 
trade-off  options. 


Probabil  ity  of 

Failure  Character 

Criteria 

Very  High 

-  The  activity  has  not  been  done  before. 

-  Technology  is  not  readily  available  to  , 

accomplish  the  activity. 

-  There  is  little  experience  in  this  type 
of  work. 

High 

-  One  or  more  major  portions  of  the  activity 
have  not  been  done  before. 

-  Experienced  people  are  not  available  to 
complete  most  of  the  activity. 

Low 

-  The  activity  has  been  done  before. 

-  People  are  available  to  complete  the 
activity. 

Very  Low 

-  The  activity  has  been  done  before. 

-  Experienced  people  are  available  to 
complete  the  activity. 

Figure  5.  PROBABILITY  OF  FAILURE 

Note:  Probability  of  Failure  is  the  reciprocal  of  the  probability  of  success. 
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Consequence 

Character 

Criteria 

Very  Severe 

A  failure  of  an  activity,  in 
itself,  that  will  cause  the  project 
to  be  cancelled. 

Severe 

A  failure,  in  combination  with  the 
failure  of  one  or  more  activities, 
that  will  cause  the  project  to  be 
cancelled . 

Moderate 

A  failure,  in  itself,  that  will 
cause  a  major  impact  on  cost, 
schedule  and/or  performance  of  the 
system/project. 

Little 

A  failure,  in  itself,  that  will 
cause  only  a  minor  impact  on  cost, 
schedule  and/or  performance. 

Very  Little 

A  failure,  In  combination  with  the 
failure  of  one  or  more  activities, 
that  will  cause  only  a  minor  impact 
on  cost,  schedule  and/or 
performance. 

Figure  6.  CONSEQUENCE  OF  FAILURE 


Trade-Off  Options.  The  trade-off 
options  are  too  numerous  to  list.  A  few 
examples  will  serve  to  highlight  selected 
options  and  provide  the  Incentive  to  gener¬ 
ate  feasible  options  within  the  scope  and 
constraints  of  redesign,  new  or  old  tech¬ 
nology,  or  redundancy.  Schedule  trade-offs 
could  Include  redesign  of  the  network  to 
permit  parallel  activities  or  add  more 
people  to  the  job.  Cost  trade-offs  could 
Include  acceptance  of  less  In  performance, 
accept  a  lesser  design,  or  extend  the  time 
required  to  accomplish  an  activity.  The 
exact  and  best  trade-off  can  only  be  made  in 
context  of  all  resources  and  limiting 
factors. 

Risk  Evaluation.  There  will  be  some 
activities  that  cannot  be  reduced  to  the 
established  Risk  Threshold.  These 
activities  must  be  evaluated  to  determine 
their  Impact  on  the  total  project.  Will  the 
failure  of  one  or  more  of  these  activities 
cause  the  project  to  fall?  Will  the  failure 
of  one  or  anore  of  these  activities  reduce 


the  system  equipment  to  a  non-usable 

system?  Will  the  failure  of  one  or  more  of 
these  activities  Increase  the  cost  to  a 
level  unacceptable  for  production?  Will  the 
failure  of  one  or  more  of  these  activities 
extend  the  engineering  development  phase  by 
one,  two,  three  or  more  >ears?  If  the 
answer  to  any  of  these  questions  Is  yes,  a 
complete  reassessment  of  the  project 

feasibility  must  be  made. 

Risk  Acceptance  and  Management.  If 
there  is  a  lesser  degree  of  risk  than  that 
outlined  in  the  paragraph  above,  the  Project 
Manager  may  decide  to  accept  and  intensively 
manage  the  high  risk  activities.  This 

management  may  include  ongoing  studies  of 
means  to  resolve  the  probability  of  failure 
or  the  consequences  of  failure  for  the 
activity.  In  addition,  periodic  reports  of 
high  risk  activities  may  be  required  which 
would  include  any  progress  In  the  area  of 
risk  reduction.  As  a  minimum,  an  individual 
should  be  appointed  to  monitor  and  report 
the  risk  reduction  progress. 
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ABSTRACT 

A  review  of  theoretical  literature  reveals  that 
most  technical  aspects  of  risk  analysis  have 
become  a  reasonably  well-defined  process  with 
many  sophisticated  techniques  in  existence.  A 
major  conclusion  from  this  review  is,  however, 
that  a  primary  need  exists  to  develop  a  better 
understanding  of  risk  analysis  in  order  to  en¬ 
hance  its  application.  Also  needed  are  better 
tools  to  enhance  use  of  both  subjective  judg¬ 
ment  and  group  decision  processes.  A  major 
concern  in  using  the  group  decision  and  consen¬ 
sus  building  techniques  seems  to  be  the  need  to 
eliminate  undue  influence  of  persuasion,  pre¬ 
vious  expressed  opinion,  majority  opinion  and 
higher  level  arm  twisting.  Many  researchers 
suggest  training  to  improve  use  of  available 
techniques  in  the  hope  that  it  would  lead  to 
increased  application  of  risk  analysis  in  the 
acquisition  process.  ^ _ _ _ 

Responses  to  a  survey  questionnaire  within  an 
Air  Force  Systems  Command  Product  Division  in¬ 
dicate  that  very  little  use  is  being  made  of 
risk  analysis  and  almost  no  use  is  being  made 
of  the  more  sophisticated  techniques.  Over 
half  of  the  respondents  had  used  PERT,  but  only 
one  had  used  VERT,  four  had  used  dynamic  model¬ 
ing  and  three  had  used  causal  integrative  mod¬ 
eling.  Where  risk  analysis  is  used  it  is  based 
mostly  on  subjective  assessment  with  group 
rather  than  individual  decisions  being  used  in 
most  work  areas.  For  those  surveyed,  engineer¬ 
ing  and  program  management  were  the  work  areas 
which  appeared  to  provide  the  most  effective 
input  for  risk  analysis.  But,  a  high  level  of 
need  exists  for  risk  analysis  in  both  sched¬ 
uling  and  cost  estimating  as  well  as  in  engi¬ 
neering  and  program  management.  A  high  level 
of  interest  was  expressed  for  formal  risk  anal¬ 
ysis  training. 


INTRODUCTION 

The  technical  aspects  of  risk  analysis  are  a 
fairly  well  understood  process  with  many  so¬ 
phisticated  techniques  in  existence  (1  and  2). 
Nevertheless,  a  review  of  the  literature  re¬ 
veals  that  practical  application  remains  at  an 
almost  nonexistent  level.  Results  of  a  survey 
on  application  of  risk  analysis  within  an  Air 
Force  Systems  Division  confirm  the  lack  of  use 
of  risk  analysis  and  provide  some  insights  in¬ 
to  potential  areas  for  increased  application. 


DESCRIPTION  OF  TECHNIQUES 

A  great  deal  of  effort  has  been  devoted  to  de¬ 
velopment  and  understanding  of  the  technical 
aspects  of  risk  analysis  and  numerous  examples 
exist  of  the  application  of  risk  analysis  to 
problems  of  a  specific  nature  and  generally 
within  a  narrow  area  of  interest.  The  first 
category  of  models  is  the  stochastic/probabil¬ 
istic  group  which  includes  PERT,  VERT  and  a 
Risk  Analysis  Model  (2,  p.  53).  The  second 
category  is  general  models.  Within  this  cate¬ 
gory,  parametric  cost  estimating  has  been  the 
primary  costing  methodology  for  DOD.  Dynamic 
modeling  is  a  third  category  and  is  based  on  a 
complex  system  of  mathematical,  models  and  works 
well  for  complex,  continuous  systems.  The  last 
category  is  the  causal  integrative  model  which 
is  used  to  determine  how  a  change  in  economic 
uncertainty  affects  the  level  of  mission,  scope 
and  funding  uncertainty. 

These  above  modeling  methodologies  are  fairly 
well-developed  and  each  of  these  methodologies 
has  been  applied  in  a  limited  sense.  A  primary 
conclusion  is  that  even  though  technical  as¬ 
pects  have  been  well-developed,  a  primary  need 
exists  to  develop  a  better  understanding  of 
risk  analysis  in  order  to  enhance  application 
of  these  methodologies  (2) .  The  review  of  lit¬ 
erature  indicates  that  only  a  limited  number  of 
individuals  involved  in  acquisition  of  major 
weapons  systems  have  been  adequately  involved 
in  application  of  risk  analysis. 

A  corollary  need  as  an  important  part  of  the 
decision  process  is  to  develop  tools  for  the 
use  of  intuition  and  subjective  judgment  (3). 
Statistical  probabilities  as  limits  of  relativ 
frequencies  of  events  and  occurrences  are  used 
routinely  in  risk  analysis  techniques.  But,  i 
many  cases,  available  probabilities  are  not 
relevant  to  a  decision  process  because  the  da+ 
base  is  outdated  or  the  current  system  does  nc 
have  a  close  counterpart  upon  which  to  base 
estimates.  Subjective  probability  then  becom 
the  valid  concept  where  probabilities  must  be 
formulated  from  the  opinion  and  experience  of 
experts  and  specialists. 

Certain  inputs  can  only  be  supplied  on  a  sub¬ 
jective  basis  by  a  select  group  of  individua. 

In  acquisition  this  group  generally  includes 
engineers,  budgeting,  pricing,  cost  estimatoi 
buyers,  contract  personnel  and  various  level 
of  program  management.  It  is  generally  be- 
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lieved  that  these  subjective  judgments  are  used 
in  almost  all  cases  throughout  the  acquisition 
organization.  However,  documentation  on  eval¬ 
uation  of  the  application  of  group  decisions 
and  use  of  consensus  building  techniques  to  de¬ 
velop  subjective  judgments  are  almost  nonesist- 
ent  (U). 

In  order  to  use  group  opinion  in  the  estimation 
process,  analysts  find  themselves  faced  with 
the  problem  of  aggregating  probability  assess¬ 
ments  of  group  opinion.  Recent  research  has 
clearly  demonstrated  that  groups  have  repeated¬ 
ly  outperformed  individuals- at  these  estimation 
and  assessment  tasks.  Fischer  found  general 
agreement  that  subjective  probability  distribu¬ 
tions  can  be  substantially  improved  by  aggre¬ 
gating  the  opinions  of  a  group  of  experts 
rather  than  relying  on  a  single  expert  (5). 
Consequently,  methodology  must  be  refined  for 
using  group  opinions  as  entering  estimates  in 
risk  and  uncertainty  analysis. 

The  combining  of  individual  judgments  into  a 
group  consensus  is  a  formidable  task.  Very 
little  information  exists  anywhere  that  di¬ 
rectly  addresses  group  behavior  in  certain  de¬ 
cision  environments  such  as  the  acquisition 
process.  Information  is  available,  however,  on 
several  behavioral  interaction  techniques.  But 
the  major  concern  in  using  these  techniques  to 
arrive  at  a  decision  is  to  eliminate  undue  in¬ 
fluence  of  persuasion,  previous  expressed  opin¬ 
ion,  majority  opinion,  higher  level  arm  twist¬ 
ing,  or  views  of  very  forceful  individuals. 

Researchers  agree  that  in  order  to  improve 
group  consensus  the  training  of  experts  spe¬ 
cialists,  decisionmakers,  and  managers  au  all 
levels  in  probabilistic  thinking  and  consensus 
building  could  lead  to  significant  improvement 
in  the  use  of  the  techniques  and  the  applica¬ 
tion  of  risk  analysis  (6).  Not  all  agree  on 
how  to  do  this  task.  Assumptions  about  exper¬ 
tise  in  a  given  subject  area  may  not  be  the  im¬ 
portant  factor  in  performance  of  a  probabilis- 
ic  task.  Maybe  the  ability  to  deal  with 
probabilistic  thought  is  what  produces  good 
probabilistic  assessments  (7). 

OVERALL  SURVEY  RESULTS 

In  order  to  evaluate  the  actual  application  of 
risk  analysis  a  survey  questionnaire  was  admin¬ 
istered  to  members  of  an  Air  Force  Systems  Com¬ 
mand  Product  Division.  Responses  were  received 
from  five  operational  units  and  represented 
five  primary  work  areas: 

work  area 
engineering 
budgeting  It  cost 

estimating  11 

program  management 

&  scheduling  10 

contracting  B  9 

pricing  5 


Of  the  LiO  respondents,  63  percent  had  received 
some  formal  training  in  statistics  and  63  per¬ 
cent  had  formal  training  in  use  of  computers. 

But,  only  30  percent  had  received  formal  train¬ 
ing  in  risk  analysis  techniques.  There  were 
two  major  exceptions  to  these  averages.  In 
program  management  all  10  persons  interviewed 
had  training  in  statistics,  9  had  training  in 
use  of  computers  and  7  had  training  in  risk 
analysis.  The  opposite  was  the  case  for  engi¬ 
neering.  Four  of  5  had  training  in  statistics 
but  only  one  had  training  in  use  of  computers 
and  none  had  training  in  risk  analysis. 

Of  the  many  available  risk  analysis  techniques 
only  the  program  evaluation  and  review  tech¬ 
nique  (PERT)  had  been  used  by  a  majority  of  the 
respondents.  Sixty  percent  had  used  PERT  but 
not  necessarily  in  their  current  work  area.  Of 
the  other  techniques,  one  had  used  the  venture  , 
evaluation  and  review  technique  (VERT),  four 
had  used  dynamic  modeling  and  three  had  used 
causal  integrative  modeling.  Obviously,  almost 
no  use  is  being  made  of  these  more  sophisti¬ 
cated  techniques. 

The  limited  use  of  the  more  sophisticated  tech¬ 
niques  was  concentrated  in  a  unit  or  a  work 
area.  In  budgeting,  two  had  used  causal  inte¬ 
grative  modeling  and  had  used  dynamic 
modeling.  The  four  using  dynamic  modeling  were 
all  in  an  airlift  and  training  group  and  the 
three  using  causal  integrative  modeling  were 
'  all  in  a  reconnaissance  and  warfare  unit.  The 
five  in  pricing  had  not  used  any  of  the  listed 
techniques . 

Most  respondents  thought  risk  analysis  was  im¬ 
portant  to  their  program  areas.  Thirty-two 
percent  thought  it  was  very  important  and  an 
additional  55  percent  said  it  was  of  average 
importance.  Nevertheless,  U5  percent  said  risk 
analysis  is  not  being  used  very  much  in  their 
program  area  even  though  most  felt  it  was  im¬ 
portant  to  their  area. 

Seventy  percent  of  the  respondents  said  the 
risk  analysis  which  was  used  in  their  work  area 
was  based  on  subjective  assessment  rather  than 
statistical  modeling  or  a  combination  of  the 
two.  All  in  engineering  and  most  in  program 
management  said  their  risk  analysis  was  based 
on  subjective  assessment.  Over  half  said  group 
decisions  rather  than  individual  decisions  were 
used  to  arrive  at  the  subjective  assessment. 

Sixty-five  percent  said  group  decisions  rather 
than  individual  decisions  were  used  for  all 
risk  decisions.  Program  management  and  engi¬ 
neering  indicated  a  high  level  of  use  of  group 
decision  prucesses  such  as  panels  of  experts. 
Sixty-three  percent  said  the  program  manager 
generally  initiated  the  group  effort.  On  the 
other  hand,  over  half  of  those  in  pricing  said 
the  group  process  was  never  used. 

Responses  on  who  has  influence  within  group  de¬ 


number  of  res 


cisions  are  fairly  consistent  with  the  pub¬ 
lished  literature.  Only  2k  percent  said  that 
group  decisions  depend  very  much  on  equal 
weight  of  all  participants.  Over  50  percent 
said  group  decisions  depend  very  much  on  undue 
influence  or  persuasion  of  one  or  more  partici¬ 
pants.  About  38  percent  said  group  decisions 
depend  very  much  on  each  of  previous  expressed 
opinion,  majority  viewpoint  and  higher  level 
arm  twisting.  Sixty  percent  said  decisions  de¬ 
pend  very  much  on  program  manager's  desires. 

An  important  factor  for  an  educational  program 
is  to  know  "how  each  unit  feels  about  its  own 
input  into  the  risk  analysis  process  as  well  as 
feelings  about  other  units'  input.  Of  the 
total  group,  li2  percent  said  engineering  pro¬ 
vided  the  most  effective  input  for  risk  analy¬ 
sis.  Another  32  percent  said  program  manage¬ 
ment  provided  most  effective  input.  Of  those 
in  program  management  7  of  10  believed  their 
own  areas  provided  the  most  effective  input. 
Four  of  5  in  engineering  said  engineering  had 
the  most  effective  input.  But,  6  of  9  in  con¬ 
tracting  said  engineering  had  the  most  effec¬ 
tive  input.  Also  only  one  of  11  in  budgeting 
said  budgeting  had  most  effective  input. 

Training  in  risk  analysis  appears  to  be  a  key 
issue  for  getting  increased  application  in  DOD. 
Most  respondents  said  a  high  level  of  need 
exists  for  risk  analysis  in  engineering  and 
program  management.  Overall  the  split  was 
about  even  between  a  high  need  and  a  low  need 
for  pricing,  budgeting  and  contracting.  In  in¬ 
dividual  areas  those  in  program  management  saw 
a  high  need  for  risk  analysis  in  all  areas  ex¬ 
cept  pricing  and  contracting.  Those  in  con¬ 
tracting  saw  a  high  need  for  risk  analysis  in 
all  areas  except  budgeting.  Those  in  engineer¬ 
ing  saw  a  high  need  in  all  areas  except  con¬ 
tracting.  Only  in  budgeting  did  they  see  a  low 
need  in  their  own  area  as  well  as  in  pricing 
and  contracting. 

A  high  level  of  interest  was  expressed  for  for¬ 
mal  risk  analysis  training.  Most  respondents 
indicated  that  education  in  procedures  to  ag¬ 
gregate  individual  preferences  and  subjective 
judgments  was  very  much  needed.  Those  in  engi¬ 
neering  saw  little  need  for  education  while 
those  in  pricing  were  split  on  the  need:  two 
of  five  said  it  was  needed  very  much  and  three 
of  five  said  not  at  all. 

Seventy-two  percent  responded  that  expertise  in 
subject  area  was  most  important  in  performance 
of  risk  assessment  and  the  others  thought  the 
ability  to  deal  with  probabilistic  thought  was 
most  important.  All  in  engineering  said  exper¬ 
tise  in  subject  area  and  three  of  five  in  pric¬ 
ing  said  ability  to  deal  with  probabilistic 
thought  was  most  important. 

WDRK  AREA  OBSERVATIONS 

ft 

The  analysis  of  specific  work  areas  reveals 


some  interesting  observations.  In  engineering 
five  of  five  gave  average  importance  (as  op- 
posed  to  very  important  or  not  very  important) 
to  risk  analysis  in  their  area  and  all  indi¬ 
cated  a  high  need  for  risk  analysis  in  their 
area.  Also,  engineering  was  identified  as  a 
most  effective  input  into  the  process.  Yet, 
three  of  five  in  engineering  said  a  low  need 
existed  for  training  in  risk  analysis  and  none 
'of  the  five  had  used  a  technique  other  than 
PERT  nor  had  they  had  any  training  in  risk 
analysis. 

In  follow-up  discussions,  engineering  work  area 
personnel  emphasized  the  need  for  a  very  clear 
distinction  between  the  traditional  risk  analy¬ 
sis  that  is  involved  in  everyday  engineering 
activities  and  the  more  recently  initiated  ef¬ 
forts  to  develop  sophisticated  program  level 
risk  analysis  techniques.  In  their  opinion, 
risk  analysis  is  already  a  part  of  formal  engi¬ 
neering  courses  and  engineers  already  have  a 
sufficient  knowledge  base  to  understand  the  re¬ 
sults  of  any  risk  assessment  that  may  be  nec¬ 
essary  as  input  into  activities  of  their  work 
area.  On  the  other  hand,  very  few  engineers 
have  a  working  knowledge  of  program-type  risk 
analysis  techniques. 

The  follow-up  discussions  disclosed  a  need  for 
a  specifically  designed  course  on  program-type 
risk  evaluation.  The  course  content  would  in¬ 
clude  an  evaluation  of  techniques  that  had  been 
successfully  applied  to  projects  which  were 
similar  to  the  systems  command  products . '  The 
course  content  should  also  include  suggestions 
as  to  whom  to  contact  for  program  risk  evalua¬ 
tion. 

Since  these  particular  engineers  are  working 
mostly  on  first  time,  unique  requirements,  they 
all  expressed  the  overriding  problem  of  identi¬ 
fying  up-front  the  big  driver  items.  In  their 
opinion,  reliance  on  experience  outweighs  use 
of  sophisticated  techniques  for  this  important 
function.  Engineers  tend  to  be  'Very  specific" 
and  therefore  are  generally  uncomfortable  with 
things  they  do  not  totally  understand  or  have 
not  previously  experienc’d.  Therefore,  with 
respect  to  the  new  techniques,  they  must  first 
see  how  the  ultimate  user  :an  make  use  of  it 
and  how  effective  it  is.  Otherwise  gut  feeling 
based  on  subjective  judgments  will  continue  to 
prevail.  They  would,  however,  like  to  Improve 
these  subjective  inputs. 

In  the  opinion  of  engineering  managsnsnt,  tech¬ 
niques  such  as  P£RT  have  been  a  burden  to  use. 
Information  from  such  techniques  is  generally 
months  behind  on-going  activities  and  any  prob¬ 
lems  have  already  been  corrected  as  they  oc¬ 
curred  and  well  before  the  information  becomes 
available.  The  techniques  end  up  being  a 
tracking  system,  not  a  decision  system. 

The  engineers  believe  a  need  exists  for  them  t 
get  back  into  cost  estimation.  The  in-depth 
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knowledge  of  the  sensitivity  of  engineering  re¬ 
quirements  is  needed  for  risk  assessment  in 
cost  estimation.  Thus,  a  better  understanding 
through  courses  or  seminars  on  risk  and  cost 
estimation  would  be  helpful.  Engineers  would 
then  have  more  confidence  in  risk  evaluation 
estimates.  An  evaluation  of  experiences  in 
parallel  development  with  other  areas  would  be 
helpful . 

The  engineers  believed  that  subjective  assess¬ 
ment  would  continue  to  depend  mostly  on  indi¬ 
vidual  experiences  but,  as  now,  v-^ntinue  to  be 
tempered  by  opinions  of  other  engineers.  In 
those  few  cases  in  engineering  where  group  con¬ 
sensus  is  sought,  a  leader  always  emerges, 
though  opportunity  already  exists  for  this  type 
consensus  building  training. 

The  respondents  in  budgeting  had  used  some  of 
the  more  sophisticated  techn^ues .  let,  as  a 
group  they  said  risk  analysis  had  little  im¬ 
portance  to  their  work  area  nor  was  it  needed, 
let,  they  saw  very  much  need  for  training  in 
risk  analysis  in  their  area.  Only  one  of 
eleven  said  budgeting  was  a  m03t  effective  in¬ 
put  into  the  process. 

In  the  opinion  of  budgeting  and  cost  estimation 
managers,  most  individuals  in  budgeting  and 
cost  estimating  have  a  sufficient  background  in 
statistics  and  business.  All  have  received 
training  in  cost  effectiveness  and  cost  esti¬ 
mating  but  most  often  risk  analysis  has  not 
been  distinguished  as  a  separate  topic.  All 
have  been  introduced  to  such  techniques  as 
learning  curves  and  quantitative  methods.  All 
have  sufficient  background  to  understand  re¬ 
sults  of  risk  analysis  and  most  would  be  com¬ 
fortable  with  use  of  techniques.  Courses  are 
available  but  the  lecturers  change  very  fre¬ 
quently.  Consequently,  not  everyone  gets  ex¬ 
posed  to  the  same  approaches.  A  seminar  series 
with  participation  by  the  many  who  work  to¬ 
gether  would  aid  in  the  use  of  available  tech¬ 
niques  . 

According  to  the  program  control  managers,  the 
available  techniques,  particularly  the  sophis¬ 
ticated  ones,,  are  not  being  used  because  of  the 
traditional  syndrome  of  wanting  to  continue 
doing  things  the  way  they  have  always  been  done 
and  of  only  allowing  a  few  days  or  a  few  weeks 
for  a  cost  estimate.  Only  when  higher  level 
decisionmakers  are  educated  to  the  need  for  use 
of  sophisticated  techniques  and  direct  the  use 
of  those  techniques  will  they  be  used.  To 
coincide  with  these  directives,  appropriate 
evaluation  time  must  be  built  into  the  system. 

Use  of  subjective  assessment  is  the  normal  pro¬ 
cedure  in  budgeting.  The  group  consensus  pro¬ 
cess  could  be  improved,  however,  if  documenta¬ 
tion  could  be  provided  on  best  procedure  for 
specific  projects  with  evaluation  of  how  suc¬ 
cessful  such  techniques  have  tjeen  for  similar 
product  groups. 


In  contracting .  three  of  eight  said  risk  analy¬ 
sis  was  very  important  to  their  area  and  the 
other  five  said  it  was  of  average  importance. 
Eight  of  the  nine  said  there  was  a  high  need 
for  risk  analysis  in  contracting  and  that 
training  in  risk  analysis  was  needed.  let, 
only  two  of  eight  had  training  in  risk  analysis 
and  they  indicated  a  limited  use  of  PERT  with 
only  one  having  used  a  more  sophisticated  tech¬ 
nique. 

In  follow-up  discussions,  contracting  managers 
insisted  that  risk  analysis  is  not  being  used 
in  contracting  because  the  need  does  not  exist. 
According  to  them,  all  risk  analysis  should 
have  been  completed  before  the  requirements  are 
submitted  to  contracting.  A  better  knowledge 
of  risk  assessment  would  be  helpful,  however, 
during  the  first  steps  of  the  negotiation  team 
process  in  which  the  core  team  (configuration, 
engineering,  product,  etc.)  all  have  input.  It 
would  be  helpful  in  the  determination  of 
•Aether  the  potential  contractor  can  meet  the 
requirements  (perform)  and  in  determining  con¬ 
tract  type  and  profit  level.  This  type  knowl¬ 
edge  should  be  provided  in  currently  available 
contracting  courses. 

As  with  other  work  areas,  time  and  cost  con¬ 
straints  generally  prevent  sophisticated  analy¬ 
sis.  Therefore  the  greatest  need  is  to  main¬ 
tain  better  the  weighted  guidelines  which  are 
now  being  used.  Up-to-date  values  and  coeffi¬ 
cients  would  help  contracting  personnel  evalu¬ 
ate  the  completed  technical  assessments  which 
are  submitted  as  requirements. 

In  pricing,  four  of  five  said  risk  analysis 
was  of  only  average  Importance  to  their  work 
area  and  three  of  five  said  there  was  a  low 
need  for  risk  analysis  in  pricing.  Consistent 
with  that  view  was  the  fact  that  only  one  of 
five  had  training  in  risk  analysis  and  none  of 
the  five  had  used  any  of  the  risk  analysis 
techniques.  The  analysis  indicates  that  either 
risk  analysis  is  not  needed  in  pricing  or  that 
the  respondents  did  not  understand  risk  analy¬ 
sis  and  therefore  were  unaware  of  its  advan¬ 
tages  as  an  aid  to  decisionmaking. 

According  to  the  pricing  work  area  managers, 
most  pricing  personnel  are  not  interested  in 
gaining  further  knowledge  in  statistics  or  risk 
analysis.  Although  most  have  the  knowledge 
bases  necessary  to  understand  what  risk  models 
are  being  used  for,  they  would  not  understand 
the  concepts  well  enough  to  evaluate  them. 
Procedures  which  are  provided  must  be  easy 
enough  for  everyone  to  use. 

Existing  models  seem  to  be  useful  in  pricing 
only  as  a  tool  to  evaluate  judgment  and  the 
output  is  good  only  until  the  first  day  of  ne¬ 
gotiation.  At  that  time  the  numbers  start 
changing  and  subjectivity  of  individual  deci¬ 
sionmakers  takes  over.  Thus,  higher  level 
management  must  understand  the  techniques  as 


only  tools  to  evaluate  Judgment  rather  than 
ends  unto  themselves. 

Hie  same  traditional  way  of  making  price  esti¬ 
mates  has  been  used  by  pricing  personnel  for 
years  and  these  personnel  reject  any  suggestion 
of  using  models  for  fear  that  the  models  will 
replace  them.  Consequently,  models  will  be 
used  only  if  mandatory  and  if  enough  time  is 
built  into  the  decision  process  to  allow  for 
their  use.  To  gain  acceptance,  models  should 
be  developed  by  working  within  the  work  area 
and  then  tested  by  outside  evaluation  before 
management  will  use  them. 

General  courses  in  risk  analysis  are  available 
to  pricing  personnel.  But,  pricing  managers 
believe  a  course  is  needed  to  show  how  the 
analysis  can  best  serve  the  work  area  an'1  it 
should  include  an  analysis  of  limitations  on 
application  of  a  given  model. 

Overall,  those  in  program  management  had  con¬ 
siderably  more  training  in  risk  analysis  and 
all  had  used  some  of  the  risk  analysis  tech¬ 
niques.  Likewise,  they  indicated  a  high  level 
of  use  of  subjective  assessment,  multiple  at¬ 
tributes,  and  group  consensus  decisions. 

In  follow-iq)  discussions  with  program  managers, 
the  view  emerged  that  overall  cost  risk  for  new 
programs  is  minimized  because  a  ceiling  has 
been  established  for  total  program  cost.  The 
scheduled  delivery  date  then  becomes  the  pri¬ 
mary  driver  of  the  program.  Each  work  unit  is 
concerned  only  with  suppo liability  of  that 
schedule  and  not  cost  risk.  . 

While  most  people  in  program  management  are 
willingly  involved  with  some  form  of  program 
risk  analysis,  tradition  is  hard  to  change  and 
managers  must  gain  confidence  in  any  new  meth¬ 
ods  or  techniques.  Higher  level  managers  must 
be  educated  on  the  validity  of  specific  models 
not  in  the  application  of  those  models.  Also, 
risk  assessment  models  seem  to  be  most  bene¬ 
ficial  in  the  software  rather  than  in  the  hard¬ 
ware  area. 

An  analysis  of  success  rates  on  use  of  models 
and  examples  of  successful  application  would  be 
more  beneficial  to  program  management  personnel 
than  actual  training  in  use  of  risk  analysis. 
But,  most  managers  want  to  guard  against  over- 
analysis  of  what  can  be  done  with  existing  ex¬ 
perience. 

In  program  management,  group  consensus  Just 
happens.  An  approach  is  put  forth  and  you  can 
stay  with  it  or  change  the  baseline.  In  group 
consensus  you  just  learn  to  use  available 
methods. 

SUMMARY 


exists  to  develop  an  awareness  and  a  better  un¬ 
derstanding  of  the  subject  in  order  to  enhance 
its  application  in  the  total  acquisition  proc¬ 
ess.  Also,  better  tools  are  needed  for  inclu¬ 
sion  of  group  decisions  and  subjective  judgment 
into  the  process. 

Survey  respondents  had  very  little  formal  train¬ 
ing  in  risk  analysis  even  though  most  had  some 
formal  training  in  statistics  and  the  use  of 
computers.  Most  respondents  recognized  the  im¬ 
portance  of  risk  analysis  to  their  work  areas 
but  the  extent  of  use  of  the  analysis  was  very 
limited.  Where  risk  analysis  was  used,  PERT 
was  the  most  often  used  technique,  although 
respondees  in  program  management  and  budgeting 
had  used  some  of  the  more  sophisticated  tech¬ 
niques. 

A  high  level  of  need  for  risk  analysis  was  per¬ 
ceived  for  engineering,  program  management  and 
the  cost  of  estimating  part  of  budgeting.  A 
high  level  of  interest  was  expressed  for  train¬ 
ing  in  risk  analysis  and  in  procedures  to  aggre¬ 
gate  individual  preferences  into  group  deci¬ 
sions. 
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The  mechanical  aspects,  of  risk  analysis  are 
fairly  well  understood.  But,  a  primary  need 
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ABSTRACT 


The  paper  describes  a  state-of-the-art  com¬ 
puter-based  project  risk  analysis  technique 
which  has  been  in  widespread  use  since  1970. 
The  technique  has  been  used  to  assist  in 
management  of  project  risks  on  over  a  hundred 
projects  worldwide,  witha  total  value  of  over 
$55  billion.  PROMAP  V*'resembles  conventional 
deterministic  project  management  tools  only  in 
that  it  uses  the  conventional  critical  path 
network  as  the  framework  for  a  project  risk 
model.  The  model  is  then  analyzed  to  deter¬ 
mine  the  interrelated  schedule  and  cost  risks 
resulting  from  time,  cost,  and  technical  per¬ 
formance  uncertainties.  At  the  same  time,  the 
model  serves  in  the  usual  way  the  routine 
functions  of  project  estimating,  planning, 
scheduling,  resourcing,  costing,  and  control. 


BACKGROUND  AND  INTRODUCTION 


In  1963  and  1964,  as  a  consequence  of  a  number 
of  significant  Defense  program  overruns,  the 
RAND  Corporation  investigated  the  analytical 
'  assumptions  of  the  PERT  project  management 
tool  which  had  been  utilized  on  an  Increasing 
number  of  DOD  programs  over  the  previous  five- 
year  period.  The  results  of  the  research  M 
revealed  that  a  major  shortcoming  of  the  criti- 
cal  path  technique  is  that  because  it  is  deter- 
mini stlc  it  does  not  adequately  account  for  the 
Impact  of  uncertainties  on  project  time  and 
cost  performance.  As  a  consequence,  PERT- 
flr.  based  schedules  and  time-related  estimates  of 
costs  are  Inherently  optimistic,  and  project 
r~  overruns  are  being  inadvertently  built  Into  the 
project  plan  from  the  very  start. 

Almost  twenty  years  later,  the  Defense  systems 
acquisition  community  now  finds  Itself  In  the 
very  peculiar  circumstance  of  continuing  to 
support  widespread  use.  of  deterministic  project 
management  tools  on  one  hand,  while  on  the 
other,  attempting  to  cope  with  the  problems  of 
Increasingly  complex  acquisition  programs  by 
sponsoring  development  of  a  proliferation  of 
special-purpose  analytical  tools  designed 
specifically  to  deal  with  project  risks  and 
uncertainties. 

T7J  See  References  5,  9,  10  and  11. 
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Project  management  has  been  employed  on  DOD 
programs  for  some  time,  but  until  recently,  it 
has  been  largely  non-rigorous.  The  original 
PERT  was  intended  to  rigorously  deal  with  un¬ 
certainties  affecting  project  time  and  cost 
performance,  and  at  that  time,  it  was  des¬ 
cribed  as  a  "stochastic"  technique  because  it 
accepts  range  estim.  *v  inputs  which  account  for 
uncertainty.  But,  the  range  values  are  re¬ 
duced  to  a  single  "expected"  value  and  the 
subsequent  critical  path  analysis  is  determi¬ 
nistic.  Hence,  the  benefits  of  probabilistic 
analysis  are  not  realized. 

At  UCLA,  In  1966,  we  started  development  of  a 
probabilistic  network  analysis  package  de¬ 
signed  specifically  for  projects  where  uncer¬ 
tainties  are  significant.  By  1968,  we  were 
applying  the  first  operational  versions  of 
PROMAP  (Project  Risk  Management  and  Planning) 
on  Navy  ship  overhaul  projects.  Early  appli¬ 
cation  results  were  reported  in  the  Navy 
Management  Review,  April/May  1969  issue. 

Navy  applications  continued  on  ship  acquisition, 
modernization,  and  repair  projects  and  in  1975, 
at  the  Fourth  Annual  DOD  Procurement  Research 
Symposium  at  Colorado  Springs,  I  described  the 
PROMAP  approach  and  compared  it  with  conven¬ 
tional  deterministic  techniques  with  the  aid  of 
explicit  results. 

The  following  year,  the  PROMAP  technique  was 
Included  In  the  Naval  Sea  Systems  Conmand  pro¬ 
ject  management  handbook.  Reef  Points,  Later 
In  1976,  I  presented  a  paper  at  the  Fifth 
Annual  DOD  Procurement  Research  Symposium  at 
Monterey,  covering  application  of  PROMAP  to 
government  contract  claims  analysis. 

Further  applications  on  DOD  and  non-Defense 
projects  in  aerospace,  energy,  transportation, 
and  construction  fields  led  to  continuing  1m* 
provement  In  the  PROMAP  approach,  and  in  1981 
as  Panel  Chairman  at  the  Air  Force  Risk  and 
Uncertainty  Workshop  at  Colorado  Springs,  I 
discussed  the  advanced  features  of  the  latest 
version,  PROMAP  V. 

Over  the  15  years  since  It  became  operational, 
the  PROMAP  V  technique  and  its  predecessors 
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have  been  successfully  applied  to  over  a 
hundred  large,  complex  Defense  and  non-Defense 
projects,  with  a  total  value  in  excess  of  $55 
billion. 


THE  PROMAP  V*  APPROACH 

The  foundation  of  the  PROMAP  V*  approach  is  the 
Project  Risk  Model  which  is  in  effect,  the 
conventional  critical  path  activity  network, 
modified  to  include  logic  and  data  accounting 
for  uncertainties  in  (see  Figure  1): 


Internal  Factors 

.  Planning  (including 
contingency  planning) 

.  Technical  performance 

.  Time  performance 

.  Resource  performance 

.  Cost  performance 

.  System  support 
readiness 


External  Factors 
.  Economic  factors 
.  Funding 

.  Environmental 
factors 

.  External 
deliverables 


By  accounting  for  the  various  types  of  uncer¬ 
tainty  in  a  single  model,  the  computer  analys 
accomplishes  the  intricate  correlations  among 
the  different  uncertainty  factors  and  schedule 
and  costs,  which  is  so  necessary  for  a  re¬ 
liable  assessment  of  project  risks.  Data  in¬ 
puts  are  accepted  from  a  variety  of  reliable 
data  sources  and  estimating  approaches,  in¬ 
cluding  empirical  data  and  parametric, 
engineering,  analogy,  factor,  and  subjective 
estimating  techniques. 

In  conducting  a  project  risk  analysis  with 
PROMAP  V*,  the  project  model  is  "run"  (simu¬ 
lated)  in  the  computer  as  many  as  several 
hundred  times;  each  run  representing  a  complet 
project  realization  from  start  to  end,  with 
activity  paths,  activity  durations,  resource 
requirements,  and  costs  sampled  from  distri¬ 
butions  contained  in  the  input  data. 

The  results  include  project  schedules  and 
schedule  risk?,  costs  and  cost  risks,  and  re¬ 
source  requirements,  together  with  data  on 
critical  activities,  diagnostics,  graphics, 
and  other  management  information. 


PROMAP  V  PROJECT  RISK  ANALYSIS 
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Figure  1.  PROMAP  V*  PROJECT  RISK  ANALYSIS 
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A.  Planning  and  Contingency  Planning 

The  conventional  critical  path  network  (pro¬ 
ject  plan),  identifies  all  project  activities 
from  start  to  completion.  Activities  are 
arranged  in  proper  sequence  of  performance,  de¬ 
picting  their  interrelations  and  interde¬ 
pendencies.  Conventional,  deterministic 
techniques  are  limited  to  representing  each 
activity  at  a  100  per  cent  likelihood  of 
,4^  occurrence. 


A  key  feature  of  the  PROMAP  V*  technique  is  its 
ability  to  account  for  uncertainty  in  the  pro¬ 
ject  plan.  For  example,  a  set  of  operational 
specifications  may  or  may  not  be  returned  to 
the  preparer  for  revisions.  Or  certain  soft¬ 
ware  design  features  may  or  may  not  be  re¬ 
jected  by  the  Project  Manager  and  returned  for 
-  .*•  modification;  a  "backup"  plan  may  be  undertaken 
to  substitute  a  less  advanced  state-of-the-art 
system  feature  should  the  primary  design  effort 
prove  to  be  unsuccessful;  or  weather  may  delay 
an  important  field  test. 

If  the  program  plan,  schedule,  costs,  and  re- 
?/-•'  source  requirements  are  to  be  realistic,  such 
uncertain  actions  must  be  accounted  for  in 
terms  of  their  likelihood  of  occurring. 


As  an  example.  Figure  2  illustrates  a  project 
plan  incorporating  a  contingency  plan  con¬ 
sisting  of  Activities  5  and  8,  representing  a 
"back-up"  in  the  likelihood  that  upon  com¬ 
pletion  of  Activity  2,  the  primary  plan  to  de¬ 
velop  an  advanced  state-of-the-art  system 
feature  (Activities  4,  7)  will  be  assessed  to 
be  too  risky.  At  the  start,  it  is  assessed 
that  the  primary  plan  has  an  8055  probability 
of  technical  success.  Accordingly,  the  "back¬ 
up"  plan  is  assigned  a  2035  probability  of  being 
implemented.  This  is  referred  to  as  "Con¬ 
tingency  Planning." 

As  another  example,  suppose  Activity  6  is  a  key 
test  of  a  critical  subassembly.  Based  on  past 
experience,  it  is  estimated  that  there  is  a  9035 
chance  that  the  test  will  be  successful 
(Activity  9)  and  a  1  035  likelihood  that  it  will 
fail  and  the  subassembly  wil  1  have  to  undergo 
some  modifications  and  retest  (Activity  10). 
This  situation  is  depicted  by  showing  Activity 
9  as  having  a  9035  probability  of  occurring, 
and  Activity  10  a  1031  probability  of  occurring. 


& 


CONTINGENCY  PLANNING 


Figure  2. 


Contingency  Planning 
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S.  Technical  Risk  Management 


Uncertainties  and  risks  inherent  in  technical 
or  software  development  can  have  major  impact 
on  the  likelihood  of  attaining  project  object¬ 
ives. 

In  the  typical  case,  the  technical  risk  ele¬ 
ments  are  identified  and  the  risks  assessed. 
Conventionally,  for  technical  risk  elements 
which  are  critical  to  project  success,  manage¬ 
ment  attention  is  directed  at  minimizing  the 
impact  on  project  performance.  With  PROMAP  V*, 
"contingency"  plans  designed  to  mitigate  the 
risks  are  developed  and  included  as  part  of  the 
overall  project  model .  Typically,  contingency 
planning  may  include  measures  such  as  early 
starts,  allocation  of  additional  resources, 
redundancy,  and  substitution  of  proven  state- 
of-the-art  technology. 

During  the  course  of  the  project,  as  the  tech¬ 
nical  development  proceeds,  periodic  reviews 
are  made  to  obtain  a  current  risk  assessment 
of  the  technical  risk  elements. 

It  is  normally  expected  that  the  risk  level  for 
an  individual  development  item  will  decrease 
as  the  work  progresses.  However,  should  the 
updated  risx  assessment  indicate  that  the 
technical  risk  level  has  not  adequately  de-  • 
creased  since  the  previous  assessment,  the 
appropriate  measures  are  taken  to  accelerate 
implementation  of  related  contingency  plans; 
the  objective  is  to  provide  assurance  that 
project  objectives  are  attained  despite  the 
continued  existence  of  technical  risks. 

C.  Schedule  Risk  Analysis 

Schedule  risk  analysis  results  include  the 
range  of  times  covering  the  span  between  the 
earliest  and  latest  possible  dates  for  project 
completion  and  Individual  milestones,  with 
accompanying  detailed-activity  schedules  and 
schedule  risks.  Figure  3  illustrates  the 
range  of  project  completion  times  for  an  ex¬ 
ample  project.  The  results  show  that  the  pro¬ 
ject  might  be  completed  in  from  240  to  330 
workdays.  The  cumulative  plot  nresents  the 
probabilities  of  project  duration  between  the 
two  extremes.  For  example,  there  Is  a  902 
probability  that  the  project  will  be  completed 
in  300  workdays  or  less,  a  60S  probability  in 
280  workdays  or  less,  and  so  forth.  The  re¬ 
sults  show  that  there  is  an  85  per  cent  chance 
of  overrunning  the  schedule  target  of  260  work¬ 
days. 


Time  Summary  Graph 

PROJECT  MOOEL  EXAMPLE  •  OVERALL  COMPLETION  TIME 


Figure  3.  Time  Sunmary  Graph 


However,  when  uncertainty  factors  are  accounted 
for,  there  can  be  a  number  of  different  net¬ 
work  paths  which  have  significant  probability 
of  becoming  critical  during  the  course  of  the 
project.  In  fact,  a  probabilistic  analysis 
will  usually  demonstrate  that  the  "longest 
time"  path  of  the  deterministic  technique  has 
significantly  less  than  a  1002  probability  of 
becoming  critical. 

For  the  sake  of  precise  project  monitoring  and 
control,  risk  analysis  focuses  on  activity 
"criticality";  the  higher  the  criticality 
value  (in  per  cent)  -  which  is  an  output  of  the 
analysis  -  the  more  sensitive  the  activity, 
with  regard  to  overall  project  schedule  per¬ 
formance. 

Accordingly, ^the  activities  requiring  close 
managerial  attention  are  those  with  the  most 
criticality.  On  some  projects,  as  many  as  35 
to  50  per  cent  of  the  project  activities  have 
a  significant  level  of  criticality. 


Activity  Criticality 

One  of  the  misleading  aspects  of  conventional 
deterministic  methods  is  the  assumption  that 
there  is  a  single  "critical"  (longest  time) 
path  which  determines  the  duration  of  the  pro¬ 
ject.  The  designated  critical  path  then  be¬ 
comes  a  focal  point  for  project  management. 


D.  Resource  Risk  Analysis 

Schedules  and  budgets  are  not  realistic  unless 
the  resources  required  to  accomplish  the  indi¬ 
vidual  project  activities  are  available  when 
needed.  Resources  may  include  personnel , 
materials,  documentation,  equipment,  facilitie 
funds,  or  suitable  environmental  conditions. 
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Most  projects  suffer  from  some  scarcity  of  re¬ 
sources — the  net  effect  can  be  to  significantly 
delay  the  project  completion  and  add  to  the 
project  cost. 

PROMAP  V*  provides  the  project  manager  with 
resource  requirements,  taking  into  account  the 
variable  start  and  finish  dates  of  the  activi¬ 
ties. 

Figures  4  and  5  show  the  difference  between 
deterministic  and  probabilistic  resource 
analyses.  The  deterministic  results  in  Figure 
4  show  a  one-day  peak  requirement  of  nine 
General  Maintenance  Men;  Figure  5  shows  a 
probabilistic  requirement  of  none  for  as  long 
as  twenty  days.  The  difference  is  due  to  the 
cumulative  impact  of  uncertainties  in  the 
probabilistic  case. 


Resource  Requirements  Graph 

DETERMMSTlC  ANALYSIS 

R***JC*  Cod*:  MAKTUAN 

Description:  GEhCRAL  MAINTENANCE.  MEN 


MAN  PROJECT  TITLE  RUN  *  OATE 

Resource  History  Graph 


R« source  Code  MAJNTMAN 
D*scriotio<T  GENERAL  maintenance  MEN 
Nunb*r  ot  Rcsouct  Unit*:  14 


Figure  5.  Probabilistic  Resource  Requirements 


E.  Cost  Risk  Analysis 

Conventionally,  cost  estimates  are  determinis¬ 
tic;  that  is,  costs  for  the  individual, line 
items  of  a  project  budget,  or  for  individual 
project  activities  of  t  network,  are  expressed 
as  single  values  representing  perhaps  the 
"best"  1./  estimate.  However,  where  there  is 
icertaTnty,  the  use  of  range  estimates  allows 
a  "cost  risk  analysis"  which  combines  the  un¬ 
certainties  for  the  different  cost  elements 
and  determines  the  overall  range  of  project 
costs  between  minimum  and  maximum  and  as  well 
as  the  risks  of  overrunning  project  cost 
targets. 


Figure  4.  Deterministic  Resource  Requirements 


1./  The  practice  varies  considerably;  "best", 
’"most  likely",  "average",  "normal",  or  no 
special  designation  at  all  may  be  given  to  the 
estimate.  On  many  projects,  there  is  no  spe¬ 
cific  standard  discipline  applied  to  the 
estimating  process. 
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Typical  results  of  a  cost  risk  analysis  are 
shown  in  Figure  6.  The  range  of  total  project 
costs  is  given  together  with  the  probabilities 
of  different  cost  outcomes  between  the  two 
extreme  values.  For  example,  it  is  shown  that 
there  is  a  60%  probability  of  expenditures 
reaching  the  amount  of  $60  million  (hence,  a 
40%  probability  of  exceeding  that  amount,  and 
by  as  much  as  $12.5  million). 


Cost  Summary  Graph 

TOTAL  PROJECT  COST  IE.OJ.  DOLLARS ) 


Figure  6.  Cost  Summary  Graph 
Total  Project  Cost  (E.O.J.  Dolla.s) 


F.  Cost/Schedule  Analysis 

Figure  7  shows  a  PROMAP  V*  cost/schedule  graph 
for  a  typical  project.  The  projection  to  com¬ 
pletion  incorporates  the  uncertainties  regard¬ 
ing  future  events  and  produces  a  "projected 
outcome  area"  which  includes  all  completion 
possibilities  between  the  extremes  in  time  and 
cost  performance.  A  specific  cost/schedule 
target  value  may  be  selected  as  representing 
any  specified  level  of  risk  acceptable  to 
management.  In  Figure  7,  the  target  value 
shown  is  the  "expected"  cost/schedule  value, 
which  has  an  average  likelihood  of  being 
realized. 


Cost  /  Schedule  Risk  Analysis 

(AT  START ) 


Figure  7.  Cost/Schedule  Projected  Performance 

Figure  8  shows  the  cost/schedule  performance 
at  a  later  stage  of  the  project.  Typically, 
the  size  of  the  projected  outcome  area  de¬ 
creases  as  certainty  replaces  uncertainty  as 
the  work  proceeds.  A  major  task  in  project 
risk  management  is  to  assure  that  the  ex¬ 
pected  value  of  the  projected  outcome  area 
does  not  materially  deviate  from  the  target 
value,  as  is  shown  in  Figure  8. 


Cost  /  Schedule  Risk  Analysis 

■  underway  ■  STAGE  I 


Figure  8.  Cost/Schedule  Risk  Analysis 


PROMAP  V*  provides  the  diagnostics  to  allow 
the  project  manager  to  make  the  specific  ad¬ 
justments  to  bring  the  projected  outcome  area 
into  an  acceptable  risk  range  (as  shown  in 
Figure  9). 


Cost  /  Schedule  Risk  Analysis  -  Adjusted 

■  UNOCRWAY"  STAGE  1 


Figure  9.  Cost/Schedule  Risk  Analysis-Adjusted 


LESSONS  LEARNED 

1.  Budgets  and  schedules  should  be  based  on 
the  results  of  risk  analyses  with  "expected" 
values  chosen  for  the  project  targets  and  the 
use  of  a  contingency  allowance  or  management 

^  reserve  should  be  avoided.  In  practice,  the 
contingency  allowance  is  set-aside  to  cover 
the  impact  of  uncertainties  on  project  cost 
performance.  However,  project  uncertainties 
generate  "minus"  as  well  as  "plus"  possibili¬ 
ties;  events  may  turn  out  better  than  expected. 
3ecause  the  conventional  contingency  allowance 
covers  just  the  "pluses",  there  is  no  planning 
to  take  advantage  of  the  "minus"  instances 
when  they  occur.  Adopting  "expected"  values 
and  contingency  planning  will  reduce  the  risks 
and  with  effective  project  risk  management, 
overrun  possibilities  will  be  minimized. 

2.  A  baseline  risk  analysis  should  be  con¬ 
ducted  early  in  the  program  and  updated  as 
appropriate  during  the  pre-award  period,  assur- 

-V  ing  the  availability  of  a  current  risk  base- 
line  at  the  time  of  source  evaluation.  , 


3.  The  system  acquisition  RFP  should  specify 
that  the  bidder  support  its  presentation  with 
the  following  data: 

a.  Identification  of  the  cost,  schedule, 
and  technical  risk  elements. 

b.  Description  of  contingency  plans  de¬ 
signed  to  reduce,  risks  on  critical  technical 
elements  to  manageable  levels. 

c.  Results  of  a  risk  analysis  covering 
cost,  schedule,  and  technical  risks. 

d.  Explanation  of  the  risk  assessment 
justification  supporting  the  bid  price. 

e.  Description  of  the  bidder's  risk 
management  plan  covering  cost,  scheduling,  and 
technical  risks,  together  with  the  details  of 
procedures  and  system  for  implementing  the 
risk  management  plan. 

4.  As  part  of  the  bid  evaluation,  the 
government  should  compare  the  bidder's  risk 
analysis  results  with  the  government's  base¬ 
line  values.  Any  significant  difference 
should  be  analyzed. 

5.  Once  the  contract  is  awarded,  the  govern¬ 
ment  should  continually  monitor  project  risks 
and  risk  trends,  the  latter  providing  a  very 
sensitive  indicator  of  problems  ahead.  The 
periodic  reviews  of  contractor  performance 
should  encompass  cost,  schedule,  and  technical 
risk  considerations  in  addition  to  the  standard 
requirements  of  DOD  I  7000.2. 
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A  LINGUISTIC  APPROACH  TO  THE  CONSTRUCTION  OF  EXPERT  SYSTEMS 


RONALD  R.  YAGER 
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NEW  ROCHELLE,  NEW  YORK  10801 

Introduction  The  set  of  data  generally  contains  the  infor¬ 

mation  particular  to  the  problem  at  hand.  A 


An  expert  system  is  a  computer  program  that 
pr.ovides  expert  level  solutions  to  important 
problems. n  In  [1]  Buchanan  and  Duda  provide 
/'anexceTlent  introduction  to  the  principles 
of  rule-based  expert  systems.  In  [2]  Buchanan 
provides  a  comprehensive  bibliography  on  ex¬ 
pert  systems.  A  particularly  well  cited  ex¬ 
ample  of  a  rule  based  expert  system  is  MYCIN 
[3,4]. 

"  Some  essential  features  required  of  such  an 
expert  system  are  seen  to  be  the  following: 

1.  It  has  the  ability  to  reason  and  to 
make  inferences  under  new  situations. 

J 

2.  It  can  use  judgmental  as  well  as  formal 
knowledge. 

j 

3.  It  provides  explanations  of  its  reason¬ 
ing  process  and  answers  questions  about  its 
knowledge^  0^,3- 

4.  It  has  the  ability  to  easily  and  modu- 
larly  add  new  knowledge. 

It  should  be  noted  that  the  first  characteris¬ 
tic  is  what  substantially  differentiates  an 
expert  system  from  an  information  system. 

The  structure  of  an  expert  system  can  be  seen 
to  consist  of  the  following  items: 

1.  A  set  of  rules. 

2.  A  set  of  data. 

3.  An  inference  mechani sm.  - - — ' 

The  set  of  rules  provide  the  knowledge  base 
of  the  expert  system.  It  is  with  these  rules 
that  one  captures  the  knowledge  of  the  expert 
in  the  computer  system.  Typically  these  rules 
are  of  the  form  of  conditional  statements  of 
the  type 

If  VX  =  A  then  *  B. 

In  this  form  Vj  and  ",2  are  variables  and  A 
and  B  are  values.  By  use  of  these  types  of 
statements  we  can  capture  the  informal  know¬ 
ledge  of  an  expert.  A  typical  example  of 
one  such  rules  would  be 

"if  the  response  type  is  rapid  then  the 
destruction  is  minimal" 

ft 


general  form  for  the  data  would  be 
V!  *  C 

The  basic  mechanism  for  reasoning  in  these 
first  generation  expert  systems  such  as  MYCIN 
is  modens  pollens. 

For  example,  from  the  rule  "if  the  response 
time  is  rapid  then  the  destruction  is  mini¬ 
mal"  and  the  data  "the  response  time  is 
rapid"  we  can  conclude  that  the  destruction 
i s  minimal . 

The  use  of  this  approach  for  reasoning  pro¬ 
vokes  at  least  two  questions 

1.  Does  the  user  and  the  designer  (expert) 
mean  the  same  thing  by  fast? 

2.  What  if  we  know  that  tte  response  time 
is  "about  five  minutes,"  can  we  say  anything 
about  the  destruction? 

Future  implementations  of  expert  systems  must 
extend  the  concepts  pioneered  by  the  develop¬ 
ers  of  MYCIN  and  other  first  generation  expert 
systems  to  include  semantics  for  natural  lang¬ 
uage  so  as  to  handle  the  above  situations. 

Fuzzy  set  and  possibility  theory  provides  a 
mechanism  for  representing  the  meaning  of 
words,  values  in  an  expert  system,  in  a 
manner  that  allows  for  the  implementation  of 
expert  systems  which  can  allow  the  necessary 
reasoning  operations  to  be  performed  in  such 
a  manner  as  to  take  into  account  the  seman¬ 
tics  of  natural  language. 

Fuzzy  Sets  in  Expert  Systems 

The  concept  of  a  fuzzy  subset  introduced  by 
L.A.  Zadeh  [5]  provides  a  generalization  of 
the  idea  of  a  set.  In  particular  it  allows 
for  the  partial  inclusion  of  elements  in 
subsets.  With  the  aid  of  this  idea  one  can 
very  naturally  represent  the  types  of  values 
associated  with  the  variables  found  in  expert 
systems.  In  particular  if  X  is  a  set  of 
elements,  a  fuzzy  subset  A  of  X  is  associated 
ciated  with  a  membership  function 

A:  X . >  [0,1] 
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such  that  A(x),  forxeX  indicates  the  degree 
to  which  x  satisfies  the  concept  defined  by 
A.  For  example,  in  our  previous  illustration 
we  could  represent  the  concepts  of  "fast"  and 
"about  five  minutes"  as  fuzzy  subsets  of  the 
real  line  representing  time. 


translates  into  the  conditional  possibility 
distribution 

TTv2|V1(x.y)  =  Min  [l,l-B(x)  +  C(y)] 

By  use  of  the  rule  of  fuzzy  compositional  in¬ 
ference  we  obtain 

TTv2(x)  =  -Max  cTTvi  (x)aTTv2  |v  (*.*)] 

xeX  c  (/\  =  min) 

Then  the  inferred  value  of  V2  is  the  fuzzy 
subset  D  such  that 

D(y)  =  TTv2(y). 

In  the  remainder  of  this  short  paper  we  would 
like  to  discuss  some  issues  related  to  the 
construction  of  expert  systems  using  fuzzy 
set  theory. 

In  particular  we  shall  briefly  discuss  four 
i ssues: 


Thus  with  the  use  of  fuzzy  subsets  a  designer 
of  an  expert  system  can  provide  semantics 
for  the  values  used  in  the  system.  Likewise 
a  user  of  such  a  system  could  provide  seman¬ 
tics  for  his  responses  to  the  systems  ques¬ 
tions.  A  typical  dialog  could  be  seen  as 
follows: 


1.  Handling  of  large  knowledge  bases 

2.  Inclusion  of  certainty/belief  associat¬ 
ed  with  knowledge 

3.  Complex  production  rules 

4.  Validation  of  expert  systems 


System:  What  is  the  response  time? 

,v  User:  Fast 

System:  What  do  you  mean  by  fast? 

User:  (at  this  point  the  user  could  draw 
'\rT|.  in  his  membership  function  for  fast) 

The  theory  of  approximate  reasoning  [6]  pro- 
vides  a  mechanism  for  making  inferences  in 
vjjita  situations  in  which  the  information  is  given 
'■JSf  in  terms  of  fuzzy  subsets.  The  rule  of  fuzzy 
compositional  inference  plays  the  analagous 
role  to  the  law  of  modens  pollens. 

Assime  we  have  the  following  situation: 

VJ  =  A 

If  Vj  •  B  then  V2  =  C, 

where  A  and  B  are  fuzzy  subsets  of  X  and  C 
is  a  fuzzy  subset  of  Y.  Using  the  theory  of 
approximate  reasoning  we  proceed  as  follows: 

Vj  «  A  translates  into  a  possibility 
distribution  such  that  TTv^(x)  *  A( x) 

The  statement 

1  f  Vi  ■  B  then  V2  =  C 


In  many  cases  of  expert  systems  the  nunber  of 
rules  necessary  to  capture  the  expert's  know¬ 
ledge  become  very  great.  In  [7]  Yager  has 
suggested  a  method  using  the  concept  of  know¬ 
ledge  trees  to  provide  an  organized  method 
for  processing  the  information  necessary  for 
an  expert  system  to  answer  any  questions. 

MYCIN  and  other  expert  systems  have  been  con¬ 
cerned  with  the  issue  of  belief  or  certainty 
factors  associated  with  the  data  and  the 
rules  in  an  expert  system. 

In  many  cases  we  have  information  of  the  type 

Vi  *  A  with  certainty 

If  Vi  =  B  then  Vp  *  C  with  a.,  certainty 

A  method  has  been  suggested  by  Yager  [8]  for 
the  inclusion  of  certainty  factors  in  a  man¬ 
ner  consistant  with  Zadeh's  theory  of  appro¬ 
ximate  reasoning.  In  particular  Yager  has 
suggested  that  the  statement 

V  »  A  with  belief  a  ,  where  aE  [0,1] 
can  be  translated  into  the  statement 

V  =  A*  (implicit  belief  one) 

where  A  and  A*  are  fuzzy  subsets  of  X  such 
that 

A*(x)  *  Min  (A(x),  o)  +  (1-a) 
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The  formulation  of  the  experts  rules  generally 
are  more  complex  than  the  simple  conditional 
statements  described  above.  A  more  represen¬ 
tative  example  of  the  types  of  rules  experts 
use  are: 

"In  at  least  about  50%  of  the  cases  if  most 
of  Vi  =  Aj,  V2  3  A2,  .  .  .  Vn  =  An  are 
satsified,  then  U  =  B." 

Methods  have  been  developed  within  the  frame¬ 
work  of  the  theory  of  approximate  reasoning 
for  the  inclusion  of  such  statements  [8]. 

An  expert  system  as  any  forecasting  type 
system  must  be  subject  to  some  validation 
procedure.  We  desire  two  properties  for  an 
expert  system  to  be  a  good  forecaster: 

1.  Correctness  of  the  answer. 

2.  Specificity  of  the  answer. 

Assume  a  system  forecasts 

V  =  B 

and  the  actual  value  is 

V  =  x* 

then  B(x*)  measures  the  degree  of  correctness 
of  the  system 

The  specificity  of  the  system  relates  to  the 
following  situation.  Assune  a  weather  fore¬ 
caster  predicts  that  the  high  termperature 
tomorrow  will  be  greater  than  10°.  While 
this  system  will  almost  certainly  have  been 
correct,  its  lack  of  being  specific  has  ren¬ 
dered  it  totally  uninformative. 

In  [9]  Yager  has  introduced  a  method  for 
measuring  the  specificity  of  a  fuzzy  subset 

1  1  da 

Sp( B)  -  f  Card  (Ba) 

0 

where  Ba  *  {x|  B(x)  a}  and 

Card  Ba  is  the  nunber  of  elements  in  Ba  . 
Conclusion 

We  feel  that  in  the  future  expert  systems 
based  upon  the  theory  of  fuzzy  subsets  will 
play  a  sigi  ificant  rol<*  In  the  decision  making 
process  invoked  in  systems  acquisition. 
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ABSTRACT 


The  thesis  of  this  paper  is  grounded  on  the  premise 
that  the  analysis  of  risk  and  uncertainty— and  ulti¬ 
mately  the  management  of  risk— can  be  meaningful 
and  effective  only  when  considered  as  an  integral 
part  of  the  decision-making  process.  Five  major 
elements  or  steps  that  encompass  the  risk  assess¬ 
ment  process— risk  identification,  risk  quantifica¬ 
tion,  risk  evaluation,  risk  acceptance  and  risk 
management— are  discussed.  The  risk  assessment 
is  shown  to  ultimately  lend  itself  to  a  multiobjec¬ 
tive  decision-making  process.  The  surrogate  worth 
trade-off' (SWT)  method  (a  multiobjective  optimiza¬ 
tion  method)  and  four  risk  assessment  method-^  \ 
ologies— the  multiobjective  statistical  method  (MSM)) 
the  partitioned  multiobjective  riskjnethcd  (PMRmTJ 
the  risk/dispersion  index  method  (RDJMv^nd  the 
uncertainty/sensitivity  index  me  th  odtUSIM  )— a  re 
L-'efly  discussed. 

1.  Introduction 


Risk  quantification— formulation  of  appropriate 
measures  of  risk  and  estimation  of  the  likeli¬ 
hood  (probability)  or  occurrence  of  all  conse¬ 
quences  associated  with  risky  events  as  well 
as  the  magnitude  of  such  consequences. 

Risk  evaluation — selection  of  an  evaluation  procedure 
(ie.g.,  optimizing  expected  value;  trade-off 
analysis)  and  analysis  of  various  possible  im¬ 
pacts  of  risky  events. 

Risk  acceptance  and  aversion— decision  making  re¬ 
garding  both  an  acceptable  level  of  risk  and 
its  equitable  distribution.  This  stage  of  risk 
assessment  also  involves  the  development  of 
risk  control  (i.e.,  methods  to  reduce  or  prevent 
risk). 

Risk  management— formulation  of  policies,  the  devel¬ 
opment  of  risk-control  options  (i.e.,  methods 
to  reduce  or  prevent  risk),  and  the  execution 
of  such  policy  options. 


The  thesis  of  this  paper  is  grounded  on  the  premise 
that  the  analysis  of  risk  and  uncertainty— and  ulti— 
mately  the  management  of  risk— can  be  meaningful 
and  effective  only  when  considered  as  an  integral 
part  of  the  decision-making  process.  To  avoid  com- 
■i  •  mon  ambiguities  of  terms  and  terminologies  associ¬ 
ated  with  this  subject,  the  following  definitions  will 
j..;  be  used— not  as  universal  definitions,  but  as  a  useful 
means  of  communicating  with  the  reader  (Haimes, 
1981). 

Risk  situations— th^se  in  which  the  potential  out- 
.  comes  can  be  described  by  reasonably  well- 
known  probability  distributions. 

Uncertainty  situations— those  in  which  potential  out¬ 
comes  cannot  be  described  in  terms  of  objec¬ 
tively  known  probability  distributions. 

Risk  assessment— a  complete  process  that  encom¬ 
passes  all  of  the  following  five  elements  or 
steps:  risk  identification,  risk  quantification, 
risk  evaluation,  risk  acceptance  and  aversion, 
and  risk  management.  The  term  "risk"  .fill  be 
generically  used  in  most  parts  of  this  paper  to 
connote  situations  of  both  risk  and  uncertainty. 

Risk  identification— identification  of  the  nature, 
types  and  sources  of  risks  and  uncertainties. 
The  end  products  of  this  stage  are  a  complete 
description  of  risky  events  and  elements  pf 
major  concern  along  with  their  causative  fac- 
>'  tors  and  mechanisms. 


The  last  two  stages  of  the  risk  assessment  process- 
risk  acceptance  and  aversion  and  risk  management- 
overlap  to  a  large  extent  and  require  the  subjective 
judgment  of  the  appropriate  decision  makers  in 
trading-off  the  noncommensurate  beneficial  and 
adverse  consequences  resulting  from  the  ultimate 
"acceptable  risk"  decision.  The  existence  of  these 
fundamental  trade-offs  among  conflicting  and  non¬ 
commensurate  multiple  objectives  and  attributes 
demands  the  consideration  of  risk  management  as 
an  integral  part  of  the  overall  decision-making  pro¬ 
cess— which  is  the  imperative  premise  assumed  in 
this  paper. 

In  summary,  from  a  multiobjective  decision-making 
perspective,  the  risk  assessment  process  consists 
of  two  major  phases  that  partially  overlap: 

a)  Information  is  quantitatively  processed  and 
evaluated  through  well-developed  procedures 
and  methodologies,  including  the  quantifica¬ 
tion  of  risk  and  uncertainty  and  the  develop¬ 
ment  of  alternative  policy  options.  The  meth¬ 
odologies  of  risk  assessment  are  the  techniques 
utilized  in  the  scientific  approach  of  esti¬ 
mating  probabilities  and  performing  risk  assess¬ 
ment  (excluding  the  explicit  application  of 
value  judgments). 

b)  Value  judgment  is  introduced,  within  the  over¬ 
all  decision-making  process,  concerning  what 
risks  and  their  associated  trade-offs  are  ac¬ 
ceptable,  what  selections  are  preferred,  what 
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policies  are  desirable,  what  constitutes  the  ul¬ 
timate  decision  (the  best-compromise  solution), 
and  what  actual  actions  should  be  taken. 

It  is  worthwhile  to  note  that  the  setting  of  value 
judgment  is  critically  important;  it  is  an  integral 
part  of  any  decision-making  process  and  thus  is  inte¬ 
gral  to  the  risk  assessment  process  itself.  This  pro¬ 
cess  also  serves  as  an  educational  medium  to  the 
decision  makers  in  their  interaction  with  the  analysts; 
it  can  help  identify  and  articulate  the  issues  upon 
which  there  is  an  agreement  among  decision  makers 
and  also  those  for  which  there  is  no  agreement;  it 
also  helps  to  make  the  implicit  explicit  (doing  this, 
however,  at  the  expense  of  embarrassing  decision 
makers  under  certain  circumstances). 


min  [fi(x),  f2(r),...,fn(x)]  (1) 

xeX 

where  X  is  the  set  of  all  feasible  solutions  defined  as 

X  =  {x  |gk(x)sO,  k  =  l,2,...,m  }  (2) 

The  functions  fj(x)  and  gfc(x)  are  well-defined  ob¬ 
jective  functions  and  constraints,  respectively,  and 
x  is  a  N-vector  of  decision  variables. 

Definition.  A  decision  x*  is  said  to  be  a  non  inferior 
solution  to  the  problem  posed  by  the  systems  (1H2) 
if  and  only  if  there  does  not  exist  another  xso  that 
fj(x)  sfj(x*),  j  =  l,2,...,n,  with  strict  inequality  hold¬ 
ing  for  at  least  one  j. 


Given  the  welter  of  theories  and  methodologies  that 
have  been  developed  over  the  last  decade  on  mul¬ 
tiple-criteria  (objective)  decision  making  (MCDM), 
"risk  management"  should  not  be  considered  a  new 
field;  rather,  it  is  a  special  case  of  MCDM— albeit  a 
very  important  one— as  well  as  an  extension  of  it. 

In  the  remainder  of  this  paper,  a  brief  summary  of 
fou-  risk  assessment  methodologies  that  have  been 
developed  within  a  multiobjective  decision-making 
framework  at  Case  Western  Reserve  University  will 
be  presented.  These  are  (i)  the  uncertainty/sensitiv¬ 
ity  index  method  (USIM)  (Haimes  et  a!.  1975  and 
Haimes  and  Hall  1977),  (ii)  the  multiobjective  sta¬ 
tistical  method  (MSM)  (Haimes  et  al.  1980),  (iii)  the 
risk/dispersion  index  method  (RDIM)  (Rarig  and 
Haimes  1983),  and  (iv)  the  partitioned  multiobjective 
risk  method  (PMRM)  (Asbeck  and  Haimes  1983). 

An  important  aspect  or  component  of  risk  manage¬ 
ment  within  a  multiobjective  decision-making  frame¬ 
work  is  that  of  solving  a  multiobjective  optimization 
problem.  The  concepts  of  Pareto  optimality  and 
trade-offs  are  so  fundamental  and  germane  here 
that  prior  to  discussing  the  above  risk  assessment 
methodologies,  we  will  concentrate  (in  the  next  sec¬ 
tion)  on  an  effective  multiobjective  optimization 
method  that  plays  a  critical  role  in  aU  of  the  above 
methods,  namely,  the  surrogate  worth  trade-off 
(SWT)  method  (Haimes  and  HaU  1974). 

2.  The  Surrogate  Worth  Trade-Off  (SWT)  Method 

The  surrogate  worth  trade-off  (SWT)  is  a  multiobjec¬ 
tive  optimization  method  (Haimes  and  Hall  1974) 
that  is  especiaUy  well  suited  for  risk  assessment. 

Fundamental  to  multiobjective  analysis  is  the  con¬ 
cept  of  the  noninferior  solution  (also  known  as 
efficient  solution  or  Pareto-optimal  solution). 
Qualitatively,  a  noninferior  solution  of  a  multiobjec¬ 
tive  problem  is  one  where  any  improvement  of  one 
objective  function  can  be  achieved  only  at  the  ex¬ 
pense  of  degrading  another. 

To  define  a  noninferior  solution  mathematically, 
consider  the  following  multiobjective  optimization 
problem  (MOP): 


The  original  version  of  the  surrogate  worth  trade-off 
method  is,  in  principle,  noninteractive  and  assumes 
continuous  variables  and  twice-differentiable  objec¬ 
tive  functions  and  constraints.  It  consists  of  four 
steps:  1)  generate  a  representative  subset  of  nor.in- 
ferior  solutions,  2)  obtain  relevant  trade-off  informa¬ 
tion  for  each  generated  solution,  3)  interact  with  the 
decision  maker  to  obtain  information  about  prefer¬ 
ence  expressed  in  terms  of  worth,  and  4)  retrieve 
the  best-compromise  solution  from  the  information 
obtained.  Extensions  of  the  surrogate  worth  trade¬ 
off  method  that  include  multiple  decision-makers, 
an  interactive  mode,  dynamic  systems,  and  nonde- 
terministir  lystems  have  been  published  in  the  liter¬ 
ature  (see,  for  example,  Chankong  and  Haimes  1983). 

The  e-constraint  approach  (Haimes  et  al.  1971), 
which  constitutes  an  important  component  upon 
which  the  SWT  is  based,  transforms  a  multiobjective 
optimization  problem  into  a  methematically  equiva¬ 
lent  single-objective  optimization  problem  by  con¬ 
verting  (n-1)  objectives  into  constraints.  The  prob¬ 
lem  resulting  from  the  e-constraint  formulation  may 


be  denoted  P^e): 

min  f|<(x) 

(3) 

subject  to 

x  eX, 

(4) 

fj(x)  S  Ej,  j  =  l,...,n,  jjik 

(5) 

Although  there  is  no  rule  to  specify  which  objective 
should  be  chosen  as  a  reference,  a  dominant  objectiv 
or  one  in  familiar  units  (such  as  dollars)  that  yields  a 
meaningful  trade-off  analysis  is  recommended. 

To  generate  an  ad  hoc  representative  subset  of  non¬ 
inferior  solutions  we  simply  select  a  reasonable  num 
ber  of  values  for  each  Ej  and  solve  P|<(e)  for  each 
combination  of  these  values  (jj<k).  The  equivalence 
between  problems  (1M2)  and  (3H5)  is  proved  in  the 
equivalence  theorem  by  Haimes  et  al.  (1971). 

The  Lagrongian  function  L(x,  x)  is  constructed  for  tl 
following  c-constraint  problem: 

min  ffc(x) 
xeX 
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subject  to 


fj(x)  s  cj,  k/j,  j=l,...,n  (6) 

where  each  e;  >  0  will  be  varied  parametrically  in 
the  process  of  constructing  the  trade-off  function. 
Form  the  generalized  Lajrangian,  L,  to  the  system 
(6): 


L(x,i)  =  fk(x)  +  jpkXkj[fj(x)  -  ej]  (7) 


where  Xkj,  j/k  are  generalized  Lagrange  multipliers. 
The  subscript  kj  in  X  denotes  that  X  is  the  Lagrange 
multiplier  associated  (in  the  e-constraint  vector 
optimization  problem)  with  the  j1^  constraint,  where 
the  objective  function  is  fk(x). 


By  considering  one  objective  function  as  primary 
and  all  others  at  minimum  satisfying  levels  as  con¬ 
straints,  the  Lagrange  multipliers  related  to  the 
(n-1)  objectives  as  constraints  will  be  either  zero  or 
positive.  The  set  of  positive  Lagrange  multipliers 
can  be  shown  to  represent  the  set  of  trade-off  ratios 
between  the  principal  objective  and  each  of  the 
constraining  objectives,  respectively.  Clearly,  these 
Lagrange  multipliers  are  functions  of  the  optimal 
level  attained  by  the  principal  objective  function  as 
well  as  the  level  of  all  other  objectives  satisfied  as 
equality  (binding)  constraints.  Consequently,  these 
Lagrange  multipliers  form  a  matrix  of  trade-off 
functions. 

In  general,  we  consider  only  those  noninferior  solu¬ 
tions  whose  associated  multipliers  xkj(x)  are  all 
strictly  positive.  In  effect,  we  have  limited  ourselves 
here  to  considering  only  solutions  that  are  proper 
and  noninferior.  In  this  case,  if  the  primary  objective 
is  changed  from  fk  to  fj,  the  trade-offs  Xu,  j/i,  can 
be  determined  from  the  Xkjs  by  using  the  formulas 

X i j  —  XjkXkj  for  all  i  ^  j  (8) 

and 


Ax  *ik  =  1/*ki  (9) 

The  decision  maker  is  supplied  with  trade-offs  and 
the  levels  of  all  corresponding  objectives.  He  then 
expresses  his  preference  for  wiether  or  not  (and  by 
how  much)  he  would  like  to  make  such  a  trade  at 
that  level.  The  surrogate  worth  function  is  then  con¬ 
structed  from  this  information. 


The  decision  maker  is  asked,  "How  (much)  would  you 
like  to  improve  fk  by  Xkj(x°)  units  per  one-unit  de¬ 
gradation  of  fj  while  all  other  objectives  remain  fixed 
at  f,  (x),  l/j,k7  Indicate  your  preference  on  a  scale 
of  +10  to  -10."  The  decision-maker's  response  is 
recorded  as  Wkj(x0),  called  the  surrogate  worth  of 
the  trade-off  between  fk  and  fj  at  x^  At  a  partic¬ 
ular  noninferior  solution,  there  will  be  n-1  questions 
to  obtain  Wk:(x°)  for  each  j/k.  In  order  to  measure 
Wfcjfx0)  on  an  interval  scale,  we  define  the  sur¬ 
rogate  worth  scale  for  measuring  Wkjs,  usingthe 
following  conventions: 


Wkj  >  0  when  X|<j  marginal  units  of  fk(x)  are 
preferred  over  one  marginal  unit  of 
fj(x),  given  the  satisfaction  of  all 
objectives  at  level  ek. 

Wkj  =  0  when  Xkj  marginal  units  of  fk(x)  are 
equivalent  to  one  marginal  unit  of 
fj(x) ,  given  the  satisfaction  of  all 
objectives  at  level  Ek. 

Wkj  <  0  when  Xkj  marginal  units  of  fk(x)  are 
not  preferred  over  one  marginal 
unit  of  fj(x),  given  the  satisfaction 
of  all  objectives  at  level  ek. 

These  conventions  define  the  scale  for  measuring 
Wkj  (Chankong  and  Haimes  1083). 

Intuitively,  one  would  expect  that  a  particular  non- 
inferior  solution  x*  is  the  best-compromise  solution 
if  the  decision  maker  is  indifferent  to  all  trade-offs 
offered  from  the  current  point  x*.  In  other  words, 
x*  is  the  best-compromise  solution  if 

Wkj(x*)  =  0  for  all  j/k  (10) 

We  can  apply  condition  (10)  to  retrieve  the  best-com¬ 
promise  solution.  If  at  some  generated  noninferior 
solution  all  the  corresponding  Wkj  vanish,  that 
generated  point  can  immediately  be  chosen  as  the 
best-compromise  solution.  Otherwise  we  use 
multiple  regression  to  construct,  for  each  j=k,  the 
surrogate  worth  function  Wkj  relating  Wkj  to  f 
for  all  *sk.  Then  the  system  of  equations 

Wkj(flvtfk-l>fk+l>”*>fn)  =  ®  (H) 

is  solved  to  determine  e*  =  (fi,...,fk_i,fk+i,...,f$T. 
The  best-compromise  solution  x*  is  then  found  by 
solving  Pk(  e*). 

The  corresponding  trade-offs  X*  = 

(Xkl»***»  Xk,k+i>...,Xkn)  can  also  be  found.  Note  that 
in  practice  there  usually  exists  an  indifference  band 
around  a  neighborhood  of  X*  within  which  the  Wkjs, 
j/k,  do  not  change.  The  decision  maker  may  be 
asked  additional  questions  to  obtain  the  indifference 
band  and  to  improve  the  accuracy  of  X*. 

3.  Risk  Assessment  Methods 

The  following  summaries  of  four  risk  assessment 
methodologies  are  taken  from  Haimes  (1983). 

3.1.  The  multiobjective  statistical  method  (MSM) 

The  multiobjective  statistical  method  (MSM)  was 
developed  for  the  U.S.  Army  Corps  of  Engineers 
(Haimes  et  al.,  1980)  to  account  for  the  risk  of 
floodings  in  the  design  and  management  of  interior 
drainage  systems.  The  method  is  an  integration  of 
multiobjective  optimization  (SWT  method)  and 
statistical  simulation  models  (Stanford-type  stream 
flow  simulation  models)  to  assess  the  probability  of 
risk  events  and  their  consequences.  The  risk  func¬ 
tions  in  the  MSM  are  first  constructed  as  functions 
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of  two  state  variables — pond  duration  arid  pond  ele¬ 
vation  of  interior  floodings.  These  two  state  vari¬ 
ables  are  then  related  to  the  system's  decision  vari¬ 
ables,  x,  using  Stanford-type  streamflow  simulation 
models.  Historical  records  associated  with  two  ran¬ 
dom  variables — precipitation  and  stream  flow — are 
then  used  to  generate  conditional  and  joint  proba¬ 
bilities  (as  appropriate)  for  the  ultimate  development 
of  the  expected  value  of  the  appropriate  risk  func¬ 
tions.  The  set  of  ordered  pairs  of  the  expected 
value  of  the  jth  risk  function,  frix1*),  j  =  1,2,...,J,  and 
its  associated  policy  decision  (xty  for  k  =  1,2,...,K, 
is  used  to  generate  the  needed  functional  relationship 
fj(x)  through  a  regression  analysis  technique.  The 
completion  of  this  last  step  yields  to  quantifiable 
risk  functions  amenable  for  optimization  via  the  SWT 
method,  where  Pareto  optimal  policies  and  their  as¬ 
sociated  trade-offs  are  generated  as  part  of  the  risk 
assessment  process. 

3.2.  The  partitioned  multiobjective  risk  method 
iPMRM) 

Central  to  the  partitioned  multiobjective  risk  method 
(PMRM)  is  the  premise  that  the  expected  value  con¬ 
cept  used  for  the  construction  of  risk  functions  suf¬ 
fers  from  the  same  flaw  associated  with  commen- 
surating  multiple  objectives  via  constant  weights 
(Asbeck  and  Haimes,  1983).  In  the  expected  value 
approach,  extreme  events  with  low  probability  of 
occurrence  are  given  the  same  proportional  weight 
or  importance  regardless  of  their  potential  cata¬ 
strophic  and  irreversible  impact.  In  the  PMRM,  on 
the  other  hand,  the  frequency  spectrum  of  the  ran¬ 
dom  variables  is  partitioned,  for  example,  to  three 
ranges— low,  medium  and  high  frequency.  Condi¬ 
tional  expected  value  risk  functions  are  then  con¬ 
structed  for  each  range  in  addition  to  the  "business 
as  usual"  unconditional  expected  value  risk  function. 
The  SWT  method  is  then  utilized  for  the  completion 
of  the  risk  assessment  process.  The  PMRM  has  been 
used  for  the  study  of  acid  rain  (Haimes,  1982). 

3.3.  The  risk/dispersion  index  method  (RD1M) 

The  heart  of  the  RD1M  is  the  construction  of  a  sen¬ 
sitivity  measure  that  accounts  for  the  effects  of 
variations  in  the  nominal  values  of  the  random  vari¬ 
ables  a.  It  is  assumed  that  aj's  are  independent  random 
variables  with  known  finite  means  and  variances.  The 
sensitivity  measure  q, called  the  dispersion  index, 
which  is  interpreted  as  a  first-order  approximation  to 
the  standard  deviation,  is  then  incorporated  in  a 
multiobjective  optimization  formulation.  The  disper¬ 
sion  index  can  also  be  interpreted  as  a  measure  of 
the  size  of  the  neighborhood  about  the  nominal  op¬ 
timal  solution  in  which  the  actual  solution  is  most 
likely  to  occur.  The  method  also  derives  a  sensi¬ 
tivity  trade-off  (when  using  the  e-constraint  formula¬ 
tion  with  the  SWT  method),  which  gives  an  explicit 
representation  of  the  trade-offs  between  the  sensi¬ 
tivity  measure  ft  and  the  other  objective  function?. 
Sirce  the  RD1M  incorporates  the  SWT  method,  it 
generates  all  needed  Pareto  optimal  solutions  to  the 
multiobjective  risk  problem  (see  Rarig  and  Haimes, 
1983). 


The  dispersion  index  is  particularly  useful  in  decision 
making.  The  information  that  Q  conveys  to  the  de¬ 
cision  maker(s)  can  be  readily  understood:  the 
larger  the  value  of  £2,  the  greater  the  possibility 
that  the  actual  solution  will  deviate  significantly 
from  the  nominal  solution.  Since  £2  is  a  scalar¬ 
valued  quantity  and  is  independent  of  the  number  of 
objectives,  any  decision  maker  who  desires  to  mini¬ 
mize  H  will  not  be  confused  by  a  deluge  of  sensi¬ 
tivity  information  that  needs  to  oe  analyzed  at  each 
prospective  solution  point  (alternative  policy  option). 

Note  that  the  fundamental  difference  between  the 
RD1M  and  the  US1M  is  that  the  former  generates  a 
sensitivity  index  on  the  basis  of  probability  distri¬ 
butions  (conditions  under  risk),  whereas  the  latter 
generates  a  sensitivity  index  without  any  reference 
to  probability  distributions(conditions  under  uncer¬ 
tainty). 

3.4.  The  uncertainty  sensitivity  index  method  (US1M) 

The  uncertainty  sensitivity  index  method  (US1M)  is 
an  effective  method  for  quantifying  uncertainty  func¬ 
tions  in  terms  of  sensitivity  indices  (see  Haimes  and 
Hal),  1977).  The  method  constructs  an  uncertainty- 
sensitivity  function  for  each  random  variable  oj 
(for  which  respective  probability  distribution  func¬ 
tions  cannot  be  assumed  to  be  reasonably  well-known) 
Such  a  function,  f j( - ),  can  take  the  following  basic 
form: 

fj(x;a)  =  [  3y(jqa)/  3aj]2|aj  =  aj  (12 

where  a  represents  the  assumed  nominal  value  of  a  ; 
and  y(- )  is  the  system's  output. 

Basically,  minimizing  the  function  fj(  - )  posed  by  eq. 
12  is  equivalent  to  minimizing  the  variation  of  the 
model's  output  [or  the  cost  function  if  fj(-)  replaces 
y(  •))  that  might  be  caused  by  the  random  variation 
of  aj.  If  eq.  12  is  considered  to  be  an  objective  func¬ 
tion  to  be  minimized  along  with  f j ( •)  and  maybe 
other  objectives,  a  vector  optimization  problem  of 
the  form  of  (1)  results.  For  example,  if  fj( - )  replace 
y(  ■)  in  eq.  12  and  the  random  vector  a  is  a  scalar  aj, 
ten  the  following  bicriterion  problem  results: 

Min  {fj(x;ai),  f2<x;ai)}  (13 

Sea 

where 

f2<x;  a  j)  =  [  3f  j(jq  aj)/3  a  =  aj  (1- 

It  has  been  shown  elsewhere  that,  for  a  given  model, 
a  major  variation  in  aj,  say  +25%,  would  yield  only 
up  to  3%  change  in  the  cost  function  fj(- )  when  the 
most  conservative  policy  x  is  followed,  but  it  would 
yield  to  as  much  as  a  50%  change  in  the  cost  functio, 
fj(  •)  when  the  business-as-usual  policy  x*  is  followe- 
(see  Haimes  et  al.,  1975;  Haimes  and  Hall,  1977). 
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A  MEASURE  OF  UNCERTAINTY  IMPORTANCE  IN  FAULT  TREES 
ALVIN  W.  ORAKE 

MASSACHUSETTS  INSTITUTE  OF  TECHNOLOGY 

(OR.  ORAKE’S  TALK  WAS  8ASED  ON  THE  FOLLOWING  OISSERTATION 
ABSTRACT  8Y  VICKI  M.  BIER  OF  M.I.T.) 


The  primary  product  of  this  work  is  the  development  of  a  measure  of  "uncertainty  importance. 
This  measure  ranks  the  components  in  a  fault  tree  in  terms  of  their  contributions  to  the  overal 
uncertainty  about  the  probability  of  system  failure.  For  sufficiently  small  fractional  change' 
in  the  variance  of  a  given  component  failure  probability,  the  uncertainty  importance  of  tha' 
component  is  defined  to  be  the  resulting  fractional  change  in  the  variance  of  the  system  failun 
prohabil ity  divided  by  the  fractional  change  in  the  variance  of  that  component’s  failure  probabi- 
1  ity. 

We  present  a  general  algorithm  for  computing  the  uncertainty  importance  of  any  component  in 
system.  This  alogritim  can  be  applied  not  only  in  the  case  where  all  component  failure  probabi 
lities  are  assumed  to  be  mutually  independent,  but  also  in  the  case  where  some  components  hav 
perfectly  correlated  failure  probabilities.  We  also  propose  a  model  for  use  in  analyzing  partia 
correlation  among  component  failure  probabilities.  Our  algorithm  is  implemented  using  MACSYMA 
a  research-oriented  programming  language  which  is  designed  to  perform  symbolic  manipulate 
of  expressions  (e.g.,  symbolic  differentiation)  as  well  as  numerical  operations. 

We  also  develop  a  simple  approximation  to  our  measure  of  uncertainty  importance.  This  approxima 
tion  is  reasonably  accurate  in  a  wide  variety  of  typical  cases,  and  can  be  computed  at  littlt 
cost  beyond  that  involved  in  estimating  the  variance  of  the  system  failure  probability.  Thf 
development  of  this  approximation  also  provides  an  improved  understanding  of  the  factors  whic 
influence  importance  of  any  given  component  depends  more  on  the  variances  of  the  failure  probabi¬ 
lities  of  the  minimal  cut  sets  in  which  that  component  appears  than  on  the  variance  of  the  compo¬ 
nent  failure  probability  itself. 

Our  work  is  illustrated  by  several  nunerical  examoles.  We  also  apply  the  techniques  which  wr 
develop  to  an  analysis  of  the  reactor  protection  (SCRAM)  system  in  a  pressurized  water  reactor. 
This  analysis  is  based  on  the  fault  tree  for  the  reactor  protection  system  presented  in  the  Reac¬ 
tor  Safety  Study  (1975).  One  result  of  our  analysis  is  an  estimate  of  rods  contributes  to  th. 
overall  uncertainty  about  the  probability  of  SCRAM  failure. 
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The  time  has  arrived  for 
accurately  predicting  programmatic 
cost  and  schedule  risk  on  large 
weapon  system  projects.  Management 
science  and  operations  research 
techniques  coupled  to  power  of 
today's  computer  provide  timely 
decision  information  for 
sophisticated  budgetary  and 
scheduling  strategies.  However,  the 
time  for  using  this  information  in  a 
systematic  fashion  has  not  arrived. 
Iris  dwiScv.5S>s  U.C  pfrabWrr, 

Some  years  ago,  I  was  working  on 
the  cost/schedule  budgets  and  risks 
for  a  major  weapon  system 
development.  I  was  suddenly  summoned 
by  the  newly  appointed  Project 
Manager.  Upon  arrival  he  promptly 
asked  "Mr.  Cockerham,  what  is  the 
probability  that  I  will  bring  this 
program  in  on  schedule?”  With 
surgical  precision,  I  replied  "Zero, 
Sir."  He  asked  "What  is  the 
probability  that  I  will  bring  this 
project  in  within  costs?"  Again,  my 
reply,  “Zero,  Sir."  He  raised  his 
voice  and  asked  "Well  then,  what  is 
the  good  news?"  I  answered,  "That  was 
the  good  news  I "  In  looking  back  at 
the  exchange,  the  Project  Manager  was 
just  trying  to  learn  something  about 
his  program  risks  and  I  was  of  the 
mind  set  to  simply  answer  the 
question  and  no  more.  This  was  a 
rather  meaningless  exchange  in  that 
probability  alone  did  not  indicate 
the  risks  nor  help  the  Project 
Manager  better  understand  his 
program. 

To  illustrate  the  point  a  risk 
profile  for  the  schedule  of  project 
is  shown  on  Figure  1. 

An  Appropriate  Statement  of  Risk: 

"There  is  a  (.5)  probability  that 
the  program  will  take  up  to  ten 
(10)  months  additional  time  to 
complete  than  planned." 


FIGURE  1 


SCHEDULE  ( MONTHS ) 

This  does  not  mean  that  the 
Planned  Schedule  should  be 
rescheduled  to  forty  eight  (48) 
months.  On  the  contrary,  the  thirty 
eight  (38)  month  Planned  Schedule  is 
used  to  drive  the  program  to  the 
earliest  possible  conclusion.  This 
strategy,  in  fact,  serves  a  useful 
purpose  but  unfortunately,  total 
program  dollars  are  often  tied  to  a 
zero  probability  schedule  which  of 
course  yields  a  zero  probability 
budget.  Such  is  just  one  of  the 
numerous  examples  of  the  difficulties 
in  communicating  with  probabilistic 
information.  Even  when  these 
difficulties  are  overcome  at  the 
program  level,  the  Project  Manager 
is,  at  best,  reluctant  to  communicate 
this  information  to  higher  levels. 
This  can  be  easily  understood  when 
one  looks  at  some  of  the  type 
information  yielded  from  a 
probabilistic  analysis.  The 
following  examples  are  true 
statements  concerning  some  wel^jL 
planned  programs. 

"Probability  of  meeting  the 
schedule  is  zero." 

"Probability  of  meeting  the  planned 
cost  is  zero.” 

"Incremental  funding  of  some  long 
lead  items  for  production  should  be 
initiated  prior  to  the  development 
program. " 


224 


"$100M  of  RDT&E  funding  is  needed 
for  the  planned  program  two  years 
after  the  IOC.” 

"There  are  negative  cost  risk  in 
the  middle  years  of  the  Engineering 
Development. 11 

“The  cost  risk  is  greater  than  the 
planned  project  budget." 

Although  there  are  inherent 
difficulties  in  communicating  risk 
information,  the  primary  difficulties 
in  DoD  are  presented  by  a  system  of 
compartmental  decision  making  that  is 
steeped  in  tradition,  power 
structures  and  resistance  to  change. 
Although  frustrating  at  times,  it  is 
recognized  that  the  aspects  of  our 
system  that  give  reasons  to  problems, 
also  give  reasons  to  much  success. 
Nevertheless,  there  are  substantial 
problems  in  implementing  risk 
information  to  the  DoD  decision 
making  process. 

The  problem  begins  at  the 
Congressional  level  in  that  there  are 
no  requirements  for  uncertainty  or 
risk  information  to  support  the 
Congressional  responsibility  of 
deciding  which  programs  are  funded 
and  how  much.  Moreover,  the 
Congressional  decisions  are 
intertwined  with  the  political 
process  which  customarily  yields 
compromised  results.  Such  decisions 
are  largely  based  upon  qualitative 
assessments  and  political  values. 
There  are  no  management  science  or 
operation  research  methods  to 
describe  the  Congressional  process. 
However,  it  is  my  contention  that  it 
is  better  to  know  the  program's 
planned  cost  and  risks  than  to  know 
only  the  program's  planned  cost. 

At  DoD  there  has  been  no 
shortage  of  words  written  and  words 
spoken  to  the  need  to  analyze,  plan 
and  budget  for  program  uncertainties. 
Lacking  in  these  words  are  firm 
instructions,  guidance,  requirements 
and  the  propensity  to  use  the 
information. 

The  Defense  Acquisition 
Initiatives  have  stimulated  some 
thinking  and  action  by  the  Services. 
However,  the  action  has  been  tenative 
and  lacking  in  application.  This  is 
understandable  in  that  there  is  no 
coordinated  push  from  DoD,  nor  has 
Congress  expressed  any  approval, 


disapproval  or  even  knowledge  of  the 
efforts  to  budget  and  plan  for  risk. 
Congress  is  hardly  to  blame  since 
there  has  been  no  DoD  spokesman  on 
the  subject  and  service  projects  mask 
the  risk  cost  in  their  budgetary 
submissions  to  Congress. 

The  following  discussion 
addresses  the  Lessons  Learned 
concerning  the  Implementation  of  the 
TRACE  Concept  for  the  Navy.  The 
lessons  learned  are  based  on 
experience  from  the  Army's  TRACE 
program  and  the  Navy's  experience 
since  September  30,  1981.  The 

information  was  generated  by  this 
author  under  a  contractual  effort  for 
the  Pilot  Application  of  the  TRACE 
Concept  for  the  Navy,  February,  1983. 
The  lessons  are  generally  applicable 
to  DoD  interest  and  are  categorized 
by  the  following  areas: 

Navy  TRACE  Implementation  Lessons 
Learned 

*  Fiscal  Management 

*  Training/Education 

*  Manpower 

*  Methodology 

*  Application 

*  Project  Management 

*  Resources 

The  nature  of  the  lesson  is 
described  as  an  observation  with 
support  rationale  and  followed  by 
recommendations .  The  subject  areas 
are  addressed  independently  but  are 
in  fact  interrelated.  Therefore,  the 
acceptance/ re jection  of  the 
observations  and  recommendations 
should  also  be  viewed  collectively. 

Fiscal  Management 

Observation : 

The  Navy  has  not  developed  the 
fiscal  management  methods  to 
systematically  incorporate  the 
elements  of  the  TRACE  concept.  There 
is  confusion  and  doubt  on  behalf  of 
Project  Managers  on  how  to  prepare 
budgetary  submissions  with  the  Risk 
Cost  Estimate  ( RCE )  and  what  adverse 
effect  may  result  when  compared  to 
previous  budgetary  submissions. 

Support : 

The  Army  developed  a  management 
system  in  conjunction  with  the 
methodology  of  the  TRACE  concept. 
The  management  system  described  the 


who,  what,  when  and  how  the  Army 
RDT&E  monies  would  be  managed. 
Organizational  infighting  and 
confusion  were  completely  avoided. 
Though  the  Army  system  has  some 
shortcomings,  the  TRACE  concept  has 
survived  largely  due  to  a 
comprehensive  management  system  from 
the  onset.  The  Navy's  approach  has 
been  overly  cautious  to  not  make 
changes  until  there  is  certainty  that 
the  TRACE  deferral  monies  will  not  be 
rejected  by  Congress.  The  Army  has 
already  provided  the  lesson  that 
Congress  will  not  remove  the  TRACE 
deferral  monies  when  properly 
presented.  However,  the  Navy  has  not 
determined  the  method  of  presentation 
nor  the  means  for  managing  the  money 
thereafter.  Navy  policy  on  these 
matters  should  not  be  mutative  and 
chance  failure  at  each  level  of 
budgetary  review.  Instead  the  lesson 
learned  by  the  Army  should  be  heeded 
and  Navy  policy  established 
accordingly  to  Navy  needs. 

Recommendation : 

Immediate  action  should  be  taken 
to  establish  the  Navy's  management 
system  of  RDTtiE  and  production  TRACE 
deferral  funds.  This  effort  should 
include  how  the  funds  are 
established,  updated,  processed, 
authorized,  expended  and  tracked. 


Training/ Education 
Observation : 

Training  and  education  at  all 
levels  in  the  Navy  is  currently 
needed . 

Support : 

The  lessons  learned  by  the  Army 

were: 

1.  TRACE  was  initially  successful 
due  to  a  comprehensive 
educational  and  training 
program.  Congressmen,  Senators, 
professional  staffers,  and  top 
management  at  DoD,  DA  and  DARCOM 
were  given  15-20  minute 
individual  presentations  on  the 
concept.  Every  Commanding 
General,  Deputy  Commander, 
Project  Manager  and  Deputy 
Project  Manager  was  individually 
given  a  20-00  minute  briefing. 
Management  staffers  at  all 
leVfels  were  briefed  in  groups  of 


twenty  (20)  or  less  for 
approximately  one  hour.  Cost 
and  system  analysts  were  giver 
two  (2)  day  courses.  All 
training  was  provided  by  the 
same  training  team  to  insure 
consistent  and  exacting 
information. 

2.  As  TRACE  was  initially 
successful  on  a  short  term  basif 
due  to  the  educational  program, 
TRACE  was  equally  unsuccessful 
on  a  long  term  basis  due  to  a 
major  extent,  to  the  lack  of 
follow-on  training. 

Recommendation : 

The  Navy  should  initiate  ar. 
intensive  training  and  educational 
program  to  introduce  the  science  and 
management  methods  associated  with 
the  TRACE  concept.  This  short  term 
training  should  be  coordinated  with  a 
Navy  handbook  and  be  similar  in  scope 
to  the  Army's  initial  training 
program.  The  training  should  be 
accomplished  in  no  more  than  six 
months  at  a  cost  of  less  than  $100K. 
Subsequent  to  the  initial  training,  a 
plan  should  be  developed  for  a  long 
term  in-house  training  capability. 
This  could  be  incorporated  into  the 
mission  of  Navy  or  DoD  schools. 


Manpower 

Observation: 

Currently,  no  significant 
manpower  capability  has  been 
manifested  within  any  service  to 
apply  the  scientific  methodology 
supporting  the  TRACE  concept . 

Support : 

From  Army,  Navy  and  major 
contractor  experience  there  is  no 
organization  or  job  function  or  job 
code  that  can  readily  be  used  to 
perform  TRACE  type  analyses.  The 
problem  is  independent  of  any  lack  of 
training  or  education  on  the  subject. 
The  problem  can  be  visualized  in  that 
virtually  all  organizations  are 
divided  functionally  in  elements, 
(i.e.,  cost,  schedule,  program  plans, 
program  management,  test,  logistics, 
quality  assurance,  procurement, 
personnel,  etc.).  However,  a  TRACE 
analysis  requires  that  detailed 
analysis  be  performed  across  all 
elements  as  relates  to  cost,  schedule 
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and  uncertainties.  Furthermore,  all 
elements  are  modeled  and  analyzed 
together.  This  is  all  possible  due 
to  the  advancements  in  computer 
hardware/software  technology. 
However,  traditional  organizational 
elements  ^re  not  structured  to  take 
advantage  of  computer  technology 
promoting  integrated  analysis  and 
decision  making. 

Recommendation :  ' 

On  a  trial  basis'  with  a  lead 
command,  the  Navy  should  detail  a 
group  of  four  to  six  individuals  to 
do  acquisition  planning.  This  group 
of  acquisition  planners  would  work  on 
multiple  projects  and  cut  across 
functional  boundaries.  Within  six  to 
twelve  months  a  credible  and  useful 
in-house  capability  for  acquisition 
planning  and  TRACE  analyses  could  be 
established. 


Methodology 


In  summary,  the  RDT5.E 
methodology  exists  and  is  available 
to  the  Navy.  The  production 
methodology  does  not  exist. 

Recommendations : 


1.  The  Navy  acquire  the 
hardware/software  for  RDT&E 
TRACE  analysis  and  make  it 
available  to  all  commands . 

2.  The  Navy  initiate  an  applied 
research  program  on  production 
cost  risk  analysis  for  a  ship, 
aircraft,  and  missile  system. 

Application 
Observation : 

The  planned  applications  for 
Navy  projects  are  insufficient  to 
support  the  implementation  of  the 
TRACE  concept. 

Support : 


Observation: 

v--V  . -  . .  —  -  -  . 


Methodologies  in  support  of 
V-  probabilistic  analysis  for  the  TRACE 
concept  have  not  been  established  by 
the  Navy. 


Support : 


For  the  analysis  and 
distribution  of  TRACE  deferral  RDT&E 
monies  the  Navy  has  used  an 
interactive  network  analyzer  known  as 
RISNET  and  methods  of  risk 
enumeration.  These  have  been 
•■~w~  accomplished  on  a  contractual  basis 
and  the  Navy  has  not  taken  steps  to 
endorse  nor  establish  an  in-house 
capability.  Having  the  methodology 
in-house  is  the  most  essential  part 
of  establishing  a  capability.  The 
computer  hardware/software  and 
operator's  instructions  must  be  made 
available  and  accessible  for  any 
significant  utilization. 


For  TRACE  production,  there  has 
been  no  effective  methodology 
developed  by  any  Service.  The  areas 
of  production  cost,  cost  overruns  and 
production  risks  are  matters  of  great 
national  concern,  regularly  voiced 
through  the  Congress  and  the  news 
,vj  media.  However,  there  is  no 
^  concerted  effort  to  develop  the 
ruethodology  to  accurately  pred'ict 
•>  production  costs  and  cost  risk. 


The  Carlucci  initiative  requires 
the  services  to  implement  TRACE  or  a 
TRACE  type  system.  The  explicit 
implication  is  that  the  TRACE  concept 
is  to  be  applied  to  all  projects  at 
all  commands.  The  Navy  has  initially 
had  good  experience  in  applying  the 
methodology,  but  only  at  one  command 
and  on  one  project.  Application  to 
programs  must  be  significantly 
increased  if  the  implementation  of 
the  TRACE  concept  is  to  be  a  serious 
consideration . 

Recommendations : 


1.  Building  on  the  NAVAIR 
experience,  the  S-3B  application 
should  be  continued  in  order  to 
demonstrate  the  maintenance  of 
the  technology  and  usefulness  on 
a  continuing  basis.  In 
addition,  two  new  applications 
should  be  initiated.  The 
projects  should  be  selected 
based  on  a  need  for  detailed 
planning  and  costing.  If 
possible,  the  projects  should  be 
of  high  complexity  and  early  in 
the  conceptual  or  development 
phase . 

2.  In  conjunction  with  lessons 
learned  in  Training/Education, 
Methodology,  Manpower  and 
Application,  the  technical 
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responsibility  for  the 
application  of  the  TRACE 
methodology  should  be  identified 
within  NAVAIR. 

3.  One  application  of  probabilistic 
network  analysis  should  be 
initiated  for  an  R&D  program  at 
each  Navy  command. 


4.  One  application  of  TRACE  for 
production  (TRACE-P)  should  be 
initiated  for  a  lead  command. 
This  would  be  an  applied 
research  effort  and  should  be 
performed  on  a  pilot  project  for 
lessons  learned. 


Project  Management 
Observations :  S-3B  Experience 

1.  S-3B  Project  experience  with 
complete  RISNET  analysis  was 
judged  favorable  and  cost 
effective . 

2.  S-3B  PMA  used  the  network  model 
as  a  vehicle  of  communication 
with  the  prime  contractor  to 
baseline  the  program  (i.e. 
program  logic,  milestones, 
deliverables,  critical  path, 
costs,  and  uncertainties).  The 
network  continues  to  provide  a 
framework  for  programmatic 
communication  between  the  PMA, 
Lockheed,  NADC,  and  JMCA. 

3.  The  prime  contractor  used  the 
network  to  better  define  the 
activities  and 
interrelationships  of  the 
program.  The  prime  contractor 
was  receptive  and  helpful  in  the 
application  of  the  RISNET 
methodology . 

4.  The  S-3B  PMA  used  the  network 
model  and  RISNET  data  to 
successfully  defend  the 
project's  baseline  budgets  and 
schedules . 

Support : 

The  TRACE  budget  determination 
is  just  one  by-product  of  interactive 
network  analysis.  The  value  to  the 
Project  Manager  encompasses  all 
aspects  of  the-  TRACE  methodology  to 
include: 


Master  Network  Display 
Sub-network  Display 
Schedules 

*  Barcharting 

*  Milestone/Deliverables 

*  Critical  and  Near  Critical 
Paths 

*  Uncertainty  .s/Risks 

*  Tracking  and  Control 
Costs 

*  Baseline  Costing  by  Fiscal  Yea 

*  Cost  Risks  by  Fiscal  Year 

*  Budget  Allocating 

,v  Multiple  Cost  Functions 
Joint  Cost  and  Schedule  Analysis 
Alternatives  and  Trade-off  Analysi 


Recommendation : 

The  Navy  should  expanc 
applications  of  the  TRACE  methodolog 
for  Project  Managers.  (See 
Application  Recommendations) . 


Resources 
Observation : 

Insignificant  resources  have 
been  committed  to  the  application  of 
the  science  embodying  the  TRACE 
concept . 

Support : 

Between  1972  and  1977  the  Arm} 
spent  $14,000  for  the  TRACE 
Guidelines.  In  1978,  $200,000  was 
used  to  purchase  RISNET  software  anc 
training  for  all  Army  RDT&E  commands. 
Since  1981,  Army  in-house 
expenditures  are  estimated  at 
$200,000  for  the  purpose  of 
establishing  a  methodology  for 
analyzing  production  cost 
uncertainties.  Since  the  Carlucc; 
Memorandum  in  April  1981,  Nav\ 
expenditures  total  approximately 
"200,000.  All  resources  expended  or 
methodology  and  training  is 
equivalent  to  approximately  5-6  mar 
years  since  1972.  This  is  less  that 
one-half  man  per  year  for  a  concep 
credited  to  save  millions  of  dollars 
Each  year  the  Government  spends  ten 
of  millions  of  dollars  or 
conferences,  seminars,  and  symposium 
to  address  the  problem  of  cos 
estimating  for  weapon  systems.  Yet 
the  most  promising  field  of  cos 
planning,  predicting  and  budgetin 
receives  virtually  no  funding  yea 
after  year.  TRACE  methodology  is  a 
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the  leading  edge  of  the  computer 
aided  decision  sciences  and  should  be 
pursued  aggressively. 

Recommendations : 

1.  DoD,  through  a  lead  Service, 
should  commit  a  minimum  of  five 
million  dollars  in  FY84/85  for 
the  ’specific  advancement  of 
TRACE  methodology  for  RDT&E  and 
production;  procurement  of 
computer  hardware  and  software; 
and  education  at  all  levels. 
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2.  DoD,  through  a  lead  Service- 
Command,  should  initiate  a  study 
to  define  a  physical  facility  of 
computers,  visual  screens, 
graphic  terminals,  plotters, 
communication  equipment  and 
software  that  would  provide 
state  of  the  art  planning, 
costing  and  control  of  programs. 
The  system  definition  should 
address  the  schedule  and 
resource  requirements  for  the 
facility,  security,  computer 
hardware,  data  base  and 
operational  software, 
documentation,  training, 
implementation  and  cost  for 
duplicate  facilities. 

Many  problems  face  the 
practical  implementation  of  risk 
information  into  the  decision 
process.  However,  the  last  year 
has  produced  greater  progress 
than  all  previous  years. 
Request  for  proposals  are 
requiring  risk  information  and 
in  at  least  one  case,  cost 
realism  was  evaluated  equal  to 
the  total  cost.  DoD  top 
management,  on  several 
occasions,  has  mentioned  the 
TRACE  concept  to  Congress  in 
testimony  regarding  the 
improvements  to  the  acquisition 
process.  Prime  contractors  are 
using  risk  methodology  to 
enhance  their  proposals  and  risk 
management  methods  to  better 
control  their  projects.  At 
least  one  of  the  services  is 
actively  reviewing  selected 
programs  explicitly  for  cost 
risk.  Most  significantly,  the 
reviews  are  conducted  by  the 
Office  of  the  Under  Secretary. 

Heretofore,  the  exclusion 
of  formal  risk  information  in 
the  decision  process  has  been 


the  greatest  shortfall  in  DoD 
and  Congressional  planning. 
Changes  have  begun  and  are 
inevitable  because  the  knowledge 
of  risk  has  been  proven  not  only 
useful  but  absolutely  necessary. 
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L  ABSTRACT 

This  paper  analyzes  tvo  work  processes 
involved  in  defense  acquisition  which  are 
replete  with  uncertainties.  These  are  the 
proposal  phase  and  the  architectural  design 
phase.  Both  phases  involve  a  vendor  design¬ 
ing  alternative  system  in  response  to  a  set 
of  stated  (and  perceived,)  requirements, 
followed  by  the  government  agency's  final 
selection  of  a  preferred  system  design  (and  a 
preferred  contractor).  Since  these  efforts 
many  times  occur  during  the  preliminary 
phases  of  the  acquisition  process,  many 
uncertainties  are  present,  including:  tech¬ 
nological  uncertainties,  uncertainties  in  the 
timely  availability  of  inputs,  workload 
uncertainties,  and  equipment  reliability  and 
maintainability,  all  of  which  lead  to  per¬ 
formance,  schedule,  and  coat  uncertainties. 

Several  issues  involving  these  uncertainties 
are  identified: 


reliability.  In  general,  we  know  how  to  deal 
with  these  factors.  This  paper  focuses  on 
certain  deficiencies  in  the  systems 
acquisition  process  itself,  which  prevent  the 
government  from  obtaining  the  most  cost- 
effective  system  meeting  the  needs  and  con¬ 
straints.  The  paper  also  presents  a  method 
of  overcoming  these  deficiencies. 

The  specific  parts  of  the  system  acquisition 
process  to  be  focused  upon  are: 

o  The  proposal  phase  from  RFP  to  Source 
Selection 

o  The  architectural  design  phase  in 
which  a  preferred  preliminary  design 
best  meeting  the  government  agency's 
needs  and  constraints  is  provided. 

The  specific  improvement  I  have  in  mind  is  an 
increased  involvement  of  the  Government 
agency  in  the  two  processes  being  treated. 


1)  Do  government  agencies  provide  vendors 
with  sufficient  information  to  enable 
them  to  design  their  most  cost-effective 
systems  with  respect  to  these  uncertain¬ 
ties? 

2)  What-  additional  information  should  be 
provided  which  will  enable  vendors  to  do 
so? 

3)  What  credible  evidence  should  vendors 
provide  in  their  proposals  and  system 
designs  which  can  increase  the  govern¬ 
ment's  confidence  that  the  system  being 
proposed  will  in  fact  be  delivered  vithin 
the  schedule  and  cost  estimated? 


Finally,  a  systems  evaluation  xethodology  is 
described  and  illustrated,  providing  a 
recommended  way  of  dealing  with  these  issues. 
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OVZEVIEV 

Defense  systems  are  composed  of  elements 
which  inherently  involve  various  uncertain¬ 
ties,  including  technological  uncertainties, 
transportation  uncertainties,  equipment  un¬ 
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This  paper  analyzes  the  work  process  involved 
in  systems  design  in  which  a  set  of  user 
requirements  and  environmental  constraints 
are  converted  into  alternative  system  designs 
and  a  preferred  design  selected.  It  identi¬ 
fies  some  basic  problems  encountered  when  a 
systems  '  design  organization  is  used  to 
initiate  this  process  of  designing  a  new 
system  for  a  client  organization.  These 
problems  fall  into  two  major  classes:  1)  how 
to  properly  state  the  requirements  and 
constraints  which  the  system  must  meet,  and 
2)  how  to  properly  evaluate  the  systems 
proposed  by  the  system  designers.  A  major 
thesis  of  this  paper  is  that  the  systems 
planning  process  is  a  cooperative  effort 
between  the  client  and  the  designer.  If  the 
latter  is  to  properly  design  a  system  he  must 
not  only  thoroughly  understand  the  require¬ 
ments,  but  also  develop  an  evaluation  pro¬ 
cedure  which  is  acceptable  to  the  client  and 
meets  his  needs. 

An  analysis  of  various  evaluation  methods 
used  is  also  provided.  The  factors  often 
used  for  evaluation  include:  1)  System 
Performance  or  other  Technical  Factors,  2) 
Date  of  Availability  of  the  System,  3)  System 


230 


Coat ,  A)  Biaka  (in  performance,  achedule  and 
coat  which  ariae  when  all  system  coaponenta 
are  not  available  "off  the  shelf,"  but  a  case 
have  to  be  developed),  and  5)  other  miscel¬ 
laneous  factora.  Centrally  the  valuta  of 
each  key  factor  for  varioua  alternativea  are 
aaaeabled  in  matrix  fora  for  validation  and 
coapariaon  purpoaea.  Unfortunately,  it  ia 
rare  that  one  alternative  ia  auperior  to  all 
othera  for  -an  deacriptora  (often  the  al¬ 
ternative  offering  auperior  performance  ia 
more  coatly  or  hat  higher  riak).  Thua  aome 
•■j  me  ana  of  relating  all  of  the  evaluation 
factora  muat  be  uaed.  Thia  ia  frequently 
done  by  applying  weighting  factora  (generally 
aelected  heuriatically) ,  which  cauaea  the 
final  "score"  to  be  highly  dependent  on  the 
valuea  of  the  veighta.  Furthermore  clienta 
often  have  difficulty  in  defending  thia  ap¬ 
proach  to  othera  requiring  auch  juatification. 

Thia  paper  examinee  in  detail  the  baaic 
proceaa  of  apecifying  requirementa,  creating 
design  alternativea,  and  evaluating  them 
againat  a  aet  of  criteria.  It  deacribea  e 
number  of  key  pitfalla  faced  by  the  systems 
deaignera  aa  veil  aa  the  evaluatora  which 
normally  occur  and  which  ahould  be  avoided 
during  a  ayatema  planning  effort.  An  im- 
.►  proved  evaluation  proceaa  avoiding  theae 
^  pitfalla  ia  preaented  for  uae  by  the  evalue- 
tion  team,  allowing  them  to  aelect  the 
preferred  alternative  in  a  more  rational, 
defenaible  faahion.  Finally,  a  method  of 
v  preaenting  evidence  which  aupporta  and 
enhancea  the  preferred  deaign  alternative  ia 
deacribed. 

While  the  main  focua  of  the  paper  ia  on  the 
architectural  deaign  proceaa  in  which  there 
y  can  be  cloae  cooperation  between  the  ayatem 
deaignera  and  the  client,  many  of  the  tech- 
niquea  described  alao  apply  to  the  ayatema 
deaign  effort  which  occurs  during  a  propoaal 
generation  effort  when  auch  cooperation  doea 
‘•'j  not  exist.  Thua  the  paper  ia  extended  to 
•is  ahov  hov  to  deal  vith  theae  problema  during  a 
propoaal  effort. 

Thia  paper  builda  on  work  in  aource  aelection 
of  EDP  ayatema  previoualy  performed  by  the 
author  for  the  Aaaiatant  Secretary  of  the  Air 
Force  (Financial  Management).  The  evaluation 
proceaa  preaented  now  includea  the  element  of 
developmental  uncertaintiea  which  vaa  not 
required  in  the  original  work.  While  the 
paper  haa  greateat  value  for  contractora  and 
Government  agenciea  involved  in  the  deaign  of 
large,  complex  ayatema  requiring  development, 
it  ia  alao  applicable  to  smaller  projects  in 
the  private  sector  aa  well. 

PART  I.  PROBLEM  DEFINITION 

PART  I  reviews  the  architectural  ayatema 
deaign  process,  indicating  tl^e  various  work 
functions  involved,  and  the  information  rs- 
'quired  by  a  ayatema  designer  if  he  ia  to  pro¬ 


vide  a  preferred  ayatem  deaign  to  a  client. 
Some  of  '  the  difficulties  in  obtaining  thia 
information  ere  deacribed. 

1.  INTRODUCTION 

A  mayor  objective  of  this  paper  ia  to 
identify  a  number  of  pitfalls  which  can 
prevent  a  systems  designer  from  proposing  and 
designing  the  moat  cost-effective  system  for 
a  client,  taking  into  account  risks  and 
uncertainties,  and  to  indicete  ways  of 
avoiding  such  pitfalls. 

At  the  heart  of  these  problema  ia  that  the 
entire  work  process  ia  generally  divided 
among  those  major  contributors,  each  of  whom 
contributes  hia  own  expertise  to  the  process: 

o  The  system  user,  who  will  operate  and 
support  the  system  once  it  ia  com¬ 
pleted,  provides  e  stetement  of  hi» 
needs  (and  desires),  es  well  as  the 
environmental  constraints  which  are 
present. 

o  The  system  designer  (generally  a  con¬ 
tractor  experienced  in  thia  area)  who 
trensletea  the  user  needs  into  a  set 
of  feasible  design  alternatives  making 
whatever  trade-offs  are  necessary,  end 
recomnends  the  preferred  system. 

o  The  procurement  organization  which 
serves  as  the  point  of  contact  with 
the  system  designer,  generally  ia 
heavily  involved  in  the  evaluation  of 
the  deaign  alternatives  in  terms  of 
the  trade-offs  of  performance,  cost, 
availability  date  and  riak,  and  makes 
the  ultimate  decision  regarding  the 
preferred  system  selected. 

For  purposes  of  this  paper  we  shall  define 
theae  participants  in  the  following  way: 

o  The  term  "Client"  will  represent  the 
procurement  organization  who  is 
funding  the  architectural  study.  The 
client  will  be  responsible  for  obtain¬ 
ing  the  specifications  from  the 
users.  Since  the  client  knows  the 
budgetary  constraints,  he  plays  a 
large  role  in  the  ultimate  systems 
evaluation  function  leading  to  the 
aelection  of  the  preferred  ayatem. 

o  The  term  "Designer"  represents  the 
systems  analysis  and  design  organiza¬ 
tion  who  haa  been  contracted  to  per¬ 
form  the  deaign  study. 

Although  a  systems  designer  works  with  the 
client  under  varying  circumstances,  thia 
paper  concentrates  on  two  disparate  situa¬ 
tions  which  bound  moat  of  the  aet.  The  first 
example  is  an  architectural  deaign  effort  in 
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which  the  designer  maintains  a  close  inter¬ 
face  with  the  client.  In  the  second  example, 
the  systems  designer  is  a  vendor  proposing  a 
system  design  to  a  client.  In  this  situation 
there  is  generally  minimal  contact  with  the 
client  during  preparation  of  the  design 
proposal. * 

2.  WORE  PROCESS  INVOLVED 

Both  situations  involve  a  common  work 
process  which  typically  includes  these 
functions  (Figure  1): 

o  Clidnt  sets  system  specifications, 
including  desired  performance,  system 
availahility  date,  and  constraints. 

o  Designer  proposes  one  or  more  design 
alternatives  potentially  meeting  the 
system  specifications. 

o  Designer  develops  an  evaluation  model 
based  on  his  understanding  of  the  job 
to  be  done  and  his  perception  of  the 
evaluation  model  to  he  used  by  the 
client. 

o  Designer  uses  the  evaluation  model  to 
evaluate  alternative  system  design 
configurations  and  selects  the 
preferred  system  design. 

o  Designer  submits  his  proposed  system 
design  to  the  client  for  his  evalu¬ 
ation. 

o  Client  validates  that  all  specifica¬ 
tions  have  been  met,  and  in  a  design 
competition,  evaluates  all  proposals 
submitted  by  vendors,  and  selects  the 
preferred  proposal. 

o  Client  makes  final  selection  of  the 
preferred  system. 


^This  situetion  is  quite  comaon  in  the 
development  of  systems  for  the  government, 
particularly  the  Department  of  Defense. 


Figure  t.  Systems  Pfenning  Work  Process 


Having  presented  an  overview  of  this  process, 
we  shall  now  examine  each  of  the  steps  in 
greater  detail,  focusing  on  some  of  the 
pitfalls  which  may  arise. 

3.  POTENTIAL  PITFALLS  IN  THE  WORK  PROCESS 

It  should  he  obvious  that  the  two  major 
drivers  of  the  design  effort  are  the  de¬ 
signer's  understanding  and  perception  of:  1) 
the  client's  specifications,  and  2)  the 

client's  evaluation  model.  Unless  the 

designer  understands  what  the  client  has 
specified,  the  final  system  design  may  he 
configured  to  produce  the  wrong  system. 
Specifically,  unless  the  designer  knows  and 
understands  the  evaluation  model  to  be  used 
by  the  client,  the  designer  will  not  he  ahle 
to  properly  make  his  performance,  cost, 
availability,  and  risk  trade-offs  to  produce 
tne  "optimal"  system  desired.  During  an 

architectural  design  effort,  there  are 

usually  opportunities  to  meet  with  the  client 
to  obtain  a  good  mutual  understanding  of  what 
the  client  really  desires  (the  "real"  system 
specifications),  as  well  as  the  proper  evalu¬ 
ation  model  which  should  be  used.  Unfortu¬ 
nately,  this  type  of  information  is  generally 
not  available  from  the  client  during  a 
proposal  effort.  Thus  in  the  next  four 
sections  we  shall  describe  a  process  for 
providing  designers  with  a  hetter  understand¬ 
ing  of  system  requirements  and  the  system 
evaluation  process,  using  the  case  of  an 
architectural  study  as  an  example.  He  shall 
then  consider  the  analogous  planning  problem 
which  should  occur  during  a  proposal  effort. 

4.  UNDERSTANDING  THE  SYSTEM  SPECIFICATIONS 

The  first  part  of  a  client's  specifica¬ 
tions  typically  describes  characteristics 
needed  by  the  designer  to  synthesize  the 
system.  Sometimes  these  descriptors  indicate 
the  concept  of  operation,  the  missions, 
functions  and  johs  to  be  done,  and  Che 
performance  characteristics  (e.g.,  speed  of 
response,  reliability  and  maintainability) 
required.  Sometimes  these  descriptors 
consist  of  a  set  of  technical  or  design 
characteristics  (e.g.,  core  size  and  number 
and  type  of  displays  for  a  data  processing 
system) . 

The  second  part  of  the  specifications  may 
consist  of  a  set  of  environmental  or  opera¬ 
tional  constraints  that  must  he  observed. 
This  set  could  include  operational  tempera¬ 
ture,  humidity,  shock  and  vibration,  as  well 
as  specifications  which  must  be  met  so  that 
this  system  can  interface  with  other  systems. 
In  addition,  the  set  of  requirements  should 
include  the  date  when  the  system  must  be 
operational. 

These  specifications  are  generally  stated  in 
two  ways.  The  first  is  a  set  of  minimum 
mandatory  requirements  that  must  be  met  or 
else  the  design  will  he  conside-ed  non- 
responsive  to  the  specifications.  Sometimes 
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Che  client  indicate*  a  desire  for  additional 
capabilities  if  they  are  available.  These 
"desirable  features"  are  not  made  part  of  the 
mandatory  requirements  since  the  designer  may 
not  be  capable  of  providing  these. 

Early  in  the  planning  effort,  the  designer 
should  review  the  system  specifications.  Any 
questions  about  these  should  be  resolved 
during  a  conference  with  the  client  early  in 
Che  design  study  effort. 

5.  UNDERSTANDING  THIS  EVALUATION  MODEL 

Having  established '*  mutual  understanding 
of  the  initial  system  "requirements"  as  the 
baseline  for  the  architectural  design  study, 
the  next  step  is  to  obtain  a  mutual  agreement 
with  Che  client  of  the  evaluation  model  to  be 
used.  Here  we  are  concerned  with  three  major 
points: 

o  The  evaluation  model  or  method  should 
be  explicit  so  that  the  designer  can 
perform  various  cost-performance 
trade-offs  to  arrive  at  a  preferred 
solution. 

o  The  evaluation  model  should  be  ra¬ 
tional,  credible  and  defensible. 

o  The  evaluation  model  should  be  agreed 
to  by  the  client.  If  Dot,  the  final 
results  obtained  may  not  be  accept¬ 
able. 

With  the  preceding  discussion  in  mind,  let  us 
now  examine  various  evaluation  methods  which 
are  used  by  various  government  agencies,  and 
describe  how  designers  may  respond  to 
each.1  This  will  be  helpful  in  determining 
whether  such  responses  are  desirable,  or  if 
the  evaluation  method  should  be  modified 
accordingly  to  produce  the  results  desired. 

5.1  The  "Extra  Performance  Is  Overkill" 

Evaluation  Method 

The  first  evaluation  method  examined, 
illustrated  in  Figure  2,  operates  as  follovs: 

o  All  evaluation  factors  and  their 
minimum  mandatory  requirements  as 
contained  in  the  Statement  of  Work 
(SOW)  are  listed  in  column  form. 

o  The  actual  values  provided  by  each 
alternative  system  being  evaluated  are 
then  listed  and  validated  by  the 
evaluator  that  these  value*  each  meet 
its  requirement. 

o  Any  system  characteristic  which  is 
over  the  minimum  level  specified  can 


^Vhile  this  discussion  specifically  applies 
to  government  contracts,  it  /Iso  applies  to 
many  non-government  contracts  as  well. 


be  considered  "overkill",  and  has  no 
additional  value  as  compared  to  the 
minimum  level. 

o  The  selection  criterion  used  is  as 
follow.'*:  choose  that  system  whose 

characteristics  individually  meet  or 
exceed  all  constraints  and  minimum 
specifications,  and  whose  Present 
Value  Life  Cycle  Cost  (PVLCC)  is  least 
among  all  alternatives  under  con¬ 
sideration. 

The  main  advantage  of  this  approach  is  that 
it  explicitly  states  the  "rules  of  the 
game".  Ideally  each  system  would  be  designed 
to  exactly  equal  the  design  specifications 
since  any  larger  values  would  generally 
result  in  higher  cost.  In  this  case  the 
evaluator  might  select  as  the  preferred 
system  alternative  the  one  which  has  the 
least  PVLCC. 

Unfortunately  system  components  come  in  dis¬ 
crete  units  rather  than  from  a  continuous 
array,  and  the  specifications  of  similar 
units  generally  differ  from  vendor  to 
vendor.  Thus  to  be  entirely  responsive  to 
the  specifications  which  have  been  issued, 
the  only  feasible  design  solution  may  be  to 
use  components  which  individually  meet  or 
surpass  the  minimum  requirements  which  have 
been  stated.  In  this  case,  excess  perform¬ 
ance  may  be  provided,  and  the  key  deficiency 
of  this  approach  is  that  extra  performance  is 
ignored.  It  should  not  be.  There  may  be 
justification  in  giving  some  extra  credit  for 


■VALUATION  MATRIX 


Validate  That  Each  Vendor  Meets  All  Minimum  Requirements 
Excess  Pert  irmance  Is  “Overklir 
Choose  by  PVLCC 

Pitfalls: 

•  Excess  Performance  Has  Worth 

*  Trade-Offs  Among  Factors  Are  Possible 

Figure  2.  E value  lion  Method  1 
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extra  performance  to  counter  balance  the 
additional  coat  which  generally  accompanies 
extra  performance. 

If  extra  credit  cannot  be  given  for  excess 
performance,  the  designer  might  be  permitted 
to  compensate  for  a  performance  deficiency  by 
providing  excess  value  of  some  related  char¬ 
acteristic.  For  example,  the  same  probabil¬ 
ity  of  kill  for  a  missile  could  be  obtained 
by  either  having  a  highly  accurate  guidance 
System  and  a  low  yield  warhead,  or  having  a 
lower  accuracy  guidance  system  and  a  higher 
yield  warhead.  Thus  it  may  be  possible  to 
achieve  the  same  end  result  at  lower  cost  to 
the  client  by  using  a  component  which  may  not 
quite  meet  an  "arbitrary”  minimum  specifica¬ 
tion  if  another  related,  higher  value  com¬ 
ponent  is  used  as  compensation.  The  designer 
can  best  make  such  determinations  since  the 
designer  usually  knows  the  relationship 
between  performance  characteristics  and  cost, 
and  hence  can  decide  which  set  of  his  avail¬ 
able  characteristics  can  best  perform  a 
defined  job  (within  constraints)  at  lowest 
total  cost. 

Thus  the  major  improvements  which  can  be  made 
to  the  "Extra  Performance  is  Overkill"  method 
are  to  define  the  jobs  to  be  done  at  the 
mission  or  functional  level  and  allow  inter¬ 
system  trade-offs  to  be  made  within  a  con¬ 
strained  set  of  boundary  conditions.  Note 
that  the  system  design  task  will  also  permit 
trade-offs  between  quality  and  quantity. 
Thus,  in  the  case  of  a  missile  system,  if  the 
probability  of  kill  of  one  missile  is  greater 
than  another,  it  may  be  possible  to  configure 
both  systems  to  achieve  a  given  level  of  tar¬ 
get  destruction,  in  which  a  lower  performance 
missile  will  require  the  use  of  more  missiles 
to  do  the  same  job  than  a  higher  performance 
missile.  The  PVLCC  calculations  will  deter¬ 
mine  the  preferred  system.  To  protect  the 
client  against  unacceptable  design  features 
(such  as  the  proposal  of  a  very  low  perform¬ 
ance  missile  in  the  previous  example),  the 
client  can  specify  a  minimum  value  that  must 
be  provided  for  any  characteristic  (such  as 
the  missile  probability  of  kill  must  exceed 
0.5). 

5.2  The  "Point  Scoring"  Evaluation  Method 

Another  disadvantage  of  the  previous 
evaluation  method  is  as  follows.  The  client 
may  have  in  mind  a  minimum  level  of  capa¬ 
bility,  but  may  desire  additional  capability 
if  obtainable  at  a  reasonable  cost.  Thus, 
some  way  must  be  found  to  give  "additional 
credit"  to  those  vendors  which  can  provide 
these  desirable  features  or  superior  char¬ 
acteristics  beyond  the  minimum  specifica¬ 
tions.  This  can  be  accomplished  by  using  the 
so-called  "point  scoring"  method,  illustrated 
in  Figure  3.  In  this  method  the  key  evalua¬ 
tion  factorg  arc  listed  again  as  one  dimen¬ 
sion  of  the  evaluation  matrix,  and  their 
values  for  each  alternative  constitute  the 


other  dimension.  As  in  the  previous  evalua¬ 
tion  method,  the  next  step  in  this  evaluation 
method  is  to  validate  that  each  of  the 
mandatory  requirements  has  been  met.  Then 
each  of  the  key  factors  where  a  value  other 
than  a  fixed  mandatory  value  is  desired  is 
assigned  two  numbers  which  will  translate  its 
value  into  a  point  score.  The  first  number 
(V  in  Figure  3)  translates  the  extra  amount 
of  performance  provided  into  a  normalized 
value  (say  from  0  to  10  to  normalize  the 
worth  of  each  factor).  The  second  number  (V. 
in  Figure  3)  provides  the  Weighting  Factor  or 
relative  worth  of  this  factor  compared  to  all 
of  the  other  factors  involved.  For  example, 
cost  may  constitute  60Z  of  the  total  score 
possible.  Choosing  the  latter  as  1000 
points,  the  value  of  W  for  cost  may  be  chosen 
as  600  points.  Thus  the  lowest  cost  design 
could  be  given  a  V  »  1,  and  each  design  would 
receive  a  V  equal  to  the  ratio  of  the  cost  of 
the  lowest  cost  design  to  the  cost  of  the 
design  under  consideration.  Thus  if  one  cost 
were  twice  as  much  as  another  alternative, 
the  lowest  cost  system  would  receive  600 
points  and  the  other  system  would  receive  300 
points.  These  numerical  values  chosen  for  V 
and  W  would  be  based  either  on  available 
operational  data  or  on  the  judgment  of  the 
technical  evaluators. 

Each  system  alternative  would  then  be  evalu¬ 
ated  with  respect  to  each  factor  in  order  to 
determine  how  many  of  the  maximum'  points 
allocated  would  go  to  each  of  the  proposed 
alternatives.  A  total  score  for  each  al¬ 
ternative  is  then  obtained  by  sunning  each  of 
its  factor  scores. 

This  method  does  have  the  advantage  of 
providing  credits  for  extra  performance. 


EVALUATION  METHOD  2 

POINT  SCORING  METHOD 
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However,  it  also  haa  several  difficulties. 
First,  while  the  key  factors  cont-ibuting  to 
the  worth  of  a  system  may  be  identified,  the 
use  of  value  and  weighting  factors  (V  and  W) 
as  the  method  of  combining  factors  is  always 
subject  to  challenge  by  other  evaluators  or 
decisionmakers.  Thus,  what  is  needed  is  a 
more  defensible  way  of  combining  the  factors 
listed. 

The  second  difficulty  inherent  in  the  point¬ 
scoring  method  is.  even  more  serious.  This 
method  combines  cost  values  with  the  tech¬ 
nical-  or  performance  values  through  the 
vehicle  of  points.  Yet  while  selecting  the 
preferred  system  based  on  highest  score  is 
intuitively  sound,  there  is  no  scientific 
justification  for  the  use  of  such  a  "figure 
of  merit"  approach.  There  are  two  more 
widely  accepted  methods  of  selecting  a 
preferred  alternative.  The  first  is  to 
select  chat  system  alternative  which  will 
perform  the  operetional  functions  and  meet 
all  constraints  at  Che  lowest  total  cost  to  a 
defined  organization  (i.e.,  pivoting  on  equal 
effectiveness).  The  second  approach  is  to 
select  Chat  alternative  which  will  yield  the 
highest  performance  of  the  operational 
functions  at  a  fixed  total  cost  (i.e., 
pivoting  on  equal  cost).  Such  a  method  must 
also  take  into  account  the  risks  and  uncer¬ 
tainties  involved. 

Lastly,  experience  has  shown  that  when  a 
large  list  of  factors  are  included  in  the 
evaluation,  the  final  score  for  each  system 
is  often  very  close  to  one  another,  rendering 
this  evaluation  method  ineffective.  One 
reason  for  this  closeness  in  score  is  because 
the  value  of  most  of  the  large  number  of 
factors  being  added  together  are  fairly  close 
to  one  another  since  most  values  correspond 
to  the  minimum  mandatory  requirements.  These 
values  overpower  value  of  the  few  remaining 
factors  which  describe  the  real  differences 


among  the  system  alternatives.  Thus,  while 
these  "matrix  evaluation  methods"  enable  the 
evaluator  to  rapidly  focus  on  the  relative 
differences  among  systems,  they  have  basic 
flaws  as  positive  selectors  of  the  preferred 
system. 

PART  II.  GENERATING  AN  IMPROVED  EVALUATION 
METHOD 

PART  II  returns  to  fundamentals  and  analyzes 
the  key  factors  which  represent  the  results 
of  an  effort  of  developing  and  constructing  a 
system  which  involves  components  that  are 
either  beyond  the  state  of  the  art  or  are  not 
readily  available  "off-the-shelf"  and  thus 
have  to  be  developed.  From  this  scenario  we 
develop  an  improved  method  for  evaluating 
proposed  system  alternatives. 

6.  CONSIDERATION  OF  DATE  OF  AVAILABILITY 
AND  ITS  UNCERTAINTY 

The  previous  discussion  of  the  two 
commonly  used  evaluation  methods  and  their 
deficiencies  concentrated  on  the  two  key 
evaluation  factors  of  system  performance 
(getting  the  jobs  done)  and  cost.  In  this 
section  we  shall  consider  two  other  evalua¬ 
tion  factors  which  must  be  considered  when 
>:he  design  contains  elements  which  must  be 
developed.  In  this  case  the  system  designer 
(and  client)  must  also  consider  technological 
or  developmental  uncertainties  which  are 
further  reflected  into: 

o  The  date  when  the  system  will  be 
available  for  use 

o  The  final  system  cost 

o  The  system  performance  achieved 

To  perform  this  analysis  we  need  to  consider 
the  entire  effort  of  developing  and  con¬ 
structing  the  system  as  a  work  process  which 
can  be  modeled  as  a  series-parallel  network 
as  shown  in  Figure  4.  This  network  indicates 


Include  All  Work  Elements:  RDT&E,  Procurement,  Installation,  Operations,  Maintenance,  Support 

Indicate  Required  Deliverables 

Indicate  Completion  Time  .and  Their  Uncertainties 


Figure  4.  Pioject  Activities  Network 
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that  this  entire  work  effort  (defined  as  "the 
project")  consists  of  a  group  of  work  activi¬ 
ties  arranged  in  a  preferred  sequence  or 
order.  Some  of  these  activities  (when 
cospleted  satisfactorily)  produce  outputs  or 
deliverables  required  as  part  of  the  State¬ 
ment  of  Work.  Each  of  the  activities  re¬ 
quires  time  and  the  expenditure  of  manpower 
and  other  resources.  Thus  using  Critical 
Path  Scheduling  techniques  the  network  can  be 
analyzed  and  the  project  completion  time  can 
be  calculated  (based  on  the  sum  of  the  times 
of  those  activities  along  the  "critical 
path").  -In  addition,  the  man-hours  required 
can  be  sunned  and  converted  into  manpower 
costs  and  total  costs. 

Having  described  the  project  effort  as  a  work 
process,  two  observations  can  be  made. 
First,  the  entire  project  effort  can  be 
completed  in  an  acceptable  fashion  only  if 
all  of  the  various  work  activities  shown  in 
figure  4  are  completed  in  the  sequence  shown, 
resulting  in  the  completion  of  the  various 
required  deliverables.  It  can  be  assumed 
that  if  an  activity  is  not  completed  satis¬ 
factorily,  the  project  effort  annot  continue 
and  it  will  be  aborted,  unless,  a  complemen¬ 
tary  activity  which  should  also  be  shown  in 
the  network  can  be  completed  as  a  substitute. 

Secondly,  given  that  an  activity  is  completed 
satisfactorily,  the  time  and  man-hours 
required  for  such  completion  can  rarely  be 
estimated  exactly  for  all  development  type 
activities.  This  uncertainty  in  completion 
time  can  best  be  represented  by  a  three-point 
estimate:  the  most  likely  value,  and  the 
limits  of  uncertainty  at  the  5th  to  the  95th 
percentile,  as  illustrated  in  figure  5a.  The 


completion  time  of  the  entire  project  will 
similarly  have  a  range  of  uncertainty  as 
illustrated  by  the  probability  distribution 
of  figures  5b  and  5c. 

6.1  Evaluation  Calculations  to  be  Made 

To  simplify  the  calculations  involved, 
j.t  will  be  assumed  that  the  project  activity 
network  constructed  describes  exactly  the 
work  process  to  be  employed.  Since  this 
assumption  will  be  applied  to  all  alterna¬ 
tives,  the  relative  accuracy  of  the  evalua¬ 
tion  should  not  be  greatly  impaired.  Here 
are  the  calculations  to  be  made: 

o  Determine  the  level  of  acceptability 
for  each  activity  in  the  network. 
Assign  the  planned  resources  to  each 
activity  and  provide  a  three-point 
estimate  of  the  time  required  to 
complete  each  activity  in  an  accept¬ 
able  fashion  using  these  resources. 

o  Estimate  the  maximum  time  each 

activity  will  be  permitted  to  con¬ 
tinue  before  the  activity,  and  hence 
the  project,  will  be  terminated.  If 
desired,  parallel  paths  can  be 

inserted  into  the  network  to  reduce 
the  chsnce  of  p'oject  termination. 

o  From  the  individual  probabilities  of 
activity  failure,  calculate  the 

probability  of  project  termination 
(failure).  Then  calculate  the  proj¬ 
ect  completion  time  as  a  probability 
distribution  when  the  project  is  suc¬ 
cessful,  as  illustrated  in  Figure  5c. 

o  Using  the  same  time  estimates, 
calculate  the  manpower  cost  and  other 


Probability 
of  Completing 
Activity 


(a)  Activity  Completion  Time 


,  _a+b+4m 
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Figure  5.  Schedule  Risk  (From  Developmental  Uncertainties) 
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costs  associated  with  each  activity 
and  the  entire  project  as  a  probabil¬ 
ity  distribution.  This  will  be  simi¬ 
lar  to  Figure  5b. 


project  in  less  than  the  maximum  acceptable 
date.  The  resulting  capabilities,  daces,  and 
total  costs  are  as  shown  in  the  three- 
dimensional,  bell-shaped  set  of  points. 
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6.2  Representing  the  Project  Results 

From  the  previous  calculations,  the  key 
evaluation  characteristics  of  the  project  may 
be  expressed  as  a  three-dimensional  probabil-r 
ity  distribution,  as  illustrated  in  figure 
6.  This  figure  should  be  interpreted  as 
illustrating  the  statistical  set  of  results 
of  performing  all  development  activities  a 
large  number  of  times.  Each  vector  (having 
one  of  the  end  points  shown)  represents  one 
of  the  results  described  by  each  of  those 
coordinates:  1)  the  system  capability  of 

meeting  the  entire  set  of  mandatory  specifi¬ 
cations,  2)  the  date  when  the  system  will 
become  available,  and  3)  the  total  cost 

required  to  obtain  the  results.  Applied  to 

these  results  are  two  threshold  levels  of 

acceptability  of:  1)  minimum  level  of  system 
capability,  and  2)  maximum  allowable  avail¬ 
ability  date.  Applying  these  levels  of 

acceptability  (to  both  L.ie  project  deliver¬ 
ables  and  to  the  system  implementation 
process  itself  as  it  progresses),  from  a 

statistical  point  of  view,  it  can  be  seen 
that  certain  of  these  "trials"  are  defined  as 
being  "unsuccessful",  since  they  do  not  meet 
the  minimum  level  of  acceptability.  These 
trials  result  in  zero  system  capability,  but 
do  consume  both  time  and  cost,  as  represented 
by  the  cluster  of  points  on  the  YZ  plan  (zero 
system  capability).  Note  that  the  times 
spent  on  the  project  vary  from  early  cancel¬ 
lation  of  the  project  to  later  cancellation. 
Costa  of  the  unsuccessful  project  "trials" 
are  also  shown.  The  other  points  of  Figure  6 
represent  the  results  of  the  successful 
"trials."  Note  that  all  successful  trials 
meet  at  least  the  minimum  level  of  system 
capability  and  all  trials  complete  the 


This  method  of  analyzing  and  evaluating 
results  shows  that  performance  risk  can  be 
defined  as  the  probability  of  meeting  the 
mini  ns  lm  set  of  requirements.  By  making  the 
assumption  that  all  activities  (of  the 
project  network)  are  independent  of  one 
another  and  each  must  be  -completed  by  some 
specified  date  (or  the  entire  project  will  be 
terminated),  the  probability  of  project 
success  can  be  calculated  as  the  joint 
probability  that  all  activities  will  be 
successful  (the  product  of  the  probabilities 
of  success  of  all  activities) .1 

Schedule  Risk  and  Cost  Risk  will  be 
treated  in  a  later  section. 

7.  DEFINING  THE  EVALUATION  OBJECTIVE 

Having  defined  the  analytical  structure 
for  the  evaluation  approach,  we  can  now 
explicitly  define  the  selection  objective  as 
follows : 

"To  select  a  proposed  system  which  performs  a 
set  of  future  required  jobs  at  given  work 
load  levels  and  meets  all  required  con¬ 
straints  including  maximum  availability  date 
at  the  lowest  total  cost,  taking  into  account 
all  uncertainties." 

This  objective  includes  the  following  three 
major  concepts: 


*Note  that  if  the  assumption  of  activity 
independence  is  not  acceptable,  a  similar, 
but  more  difficult,  analysis  can  be  made 
taking  the  pertinent  dependencies  into 
account. 
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a.  The  contractor  must  show  that  each 
of  his  system  alternatives  can  perform  all  of 
the  future  joba  and  meet  all  the  constraints. 

b.  Lowest  total  present  value  life 
cycle  cost  should  be  the  selection  criterion. 

c.  Development  and  job  uncertainties 
are  the  key  factors  which  make  the  evaluation 
selection  process  a  difficult  one. 

8.  SUMMARY  OF  SYSTEMS  PLANNING  APPROACH 

Figure  7  summarizes  the  steps  to  be 
followed  in  an  architectural  design  study: 

a.  The  client  will  define  one  or  more 
sets  of  mandatory  system  requirements  in 
terms  of  jobs  to  be  done  and  constraints  to 
be  met.  The  client  will  also  specify  a  set 
of  desirable  features  over  and  above  the 
mandatory  requirements  which  they  would  like 
to  obtain,  if  possible,  and  a  sec  of  jobs 
which  would  use  such  desirable  features. 

b.  The  designer  will  design  a  number  of 
system  alternatives  which  meet  each  set  of 
mandatory  requirements,  on  or  before  a  speci¬ 
fied  availability  date,  at  a  level  of  risk 
specified  by  the  client. 

c.  The  designer  will  calculate  the 
present  value  life  cycle  cost  (PVLCC)  of 
meeting  the  stated  set  of  requirements  at  a 
level  of  risk  specified  by  the  client. 

d.  The  designer  will  also  provide  total 
cost  data  relevant  to  providing  and  operating 
each  desirable  feature  he  provides,  as  em¬ 
bedded  in  the  representative  jobs  for  which 
the  desirable  feature  is  to  be  used. 

e.  Based  on  this  data,  the  designer 
will  calculate  the  cost  of  performing  the  set 


of  jobs  associated  with  each  Desirable  Fea¬ 
ture  proposed  by  the  designer.  This  cost 
will  be  compared  against  the  cost  of  perform¬ 
ing  the  set  of  jobs  if  the  proposed  Desirable 
Feature  were  not  available.  For  each  of 
these  sets  of  jobs,  the  least  costly  way  of 
performing  these  jobs  will  be  chosen  snd  this 
cost  added  to  the  cost  of  performing  the 
mandatory  jobs. 

f.  These  results  (the  preferred  system 
for  each  level  of  system  capability)  will  be 
shown  to  the  client. 

g.  The  client  will  select  the  final 
preferred  system  based  on  a  comparison  of  the 
incremental  cost  to  the  incremental  gain  for 
increasing  levels  of  system  capability. 

PAST  III.  APPLYING  EVALUATION  METHOD  IN  AN 
AECHITECTUEAL  DESIGN  STUDY 

PAST  III  amplifies  the  description  of  the 
design  approach  by  shoving  how  to  apply  the 
approach  in  an  architectural  design  study. 

9.  AN  EXAMPLE  OF  THIS  PROPOSED  APPROACH 

Having  described  the  approach  to  be 
followed,  we  shall  now  consider  an  example  of 
how  the  approach  would  operate  in  practice. 
The  example  used  is  that  of  an  architectural 
design  study  of  an  information  system. 

9.1  Client  Issues  the  Total  Set  of  Require¬ 
ments 

As  mentioned  previously,  this  would 
include: 

o  The  basic  system  workload  (as  char¬ 
acterized  by  a  representative  set  of 
EDP  jobs),  in  terms  of  the  mandatory 


1.  Client  will  specify  alternative  sets  of  design  requirements  in 
terms  of  jobs  to  be  done,  performance  and  constraints. 

2.  Client  will  specify  minimum  mandatory  requirements  and 
extra  features  desired. 

3.  Constrain  designers  to  provide  alternative  dssigns  which 
meet  the  set  of  system  requirements,  including  maximum 
availability  date  and  level  of  risk. 

4.  For  each  alternative,  calculate  the  present  value  life-cycle 
cost  (PVLCC). 

5.  Determine  the  worth  of  any  significant  extra  performance. 

6.  Select  the  system  which  meets  the  set  of  requirements 
and  constraints  at  lowest  PVLCC  to  client  while  accounting 
for  risks  and  uncertainties. 

7.  Show  preliminary  results  to  client  and  trade-off  specifications 
with  feasible  system  results. 

8.  Designer  assists  the  client  in  finalizing  on  Preferred  System. 


Figure  7.  Summery  or  Architectural  Design  Approach 
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capabilities  Co  be  met  over  time,  as 
illustrated  in  Figure  8 

o  All  mandatory  constraints  to  be  met 

o  A  statement  of  Desirable  Features  (or 
extra  capability  desired),  and  a 
statement  of  the  set  of  jobs  for 
which  each  desirable  feature  would  be 
used  if  provided  by  the  designer 

9.2  Consideration  of  Mandatory  System 

Capability 

The  first  step  which  the  designer  mu'.t 
take  is  to  configure  one  or  more  system 
alternatives  which  will  meet  or  exceed  the 
minimum  mandatory  workload  requirement. 
Figure  8a  shows  the  "input  demand  function" 
(in  this  case  a  workload  which  will  be 
increasing  over  time).  The  increase  shown  is 
expected  to  be  gradual  from  start  until  tj, 
when  a  large  increase  is  expected.  The 
workload  then  continues  to  increase  gradually 
until  t2.when  a  second  large  increase  will 
occur.  After  C2  the  increase  is  again 
gradual.  The  planned  system  life  is  five 
years,  which  occurs  at  t3< 


In  the  example  being  presented,  the  objective 
of  the  system  is  to  provide  sufficient  capa¬ 
bility  to  process  the  forecasted  daily 
workload  within  a  24-hour  period.  However, 
as  shown  in  Figure  8b,  the  24 -hour  period 
must  also  include  time  for  Rework  to  correct 
all  errors  detected,  and  Down  Time  (for  both 
preventive  and  corrective  maintenance).  Vith 


.System 

Capability 


Figure  8s.  Meeting  the  Workload 


Down  Tima 


Work  Load 


Total  Tima 
•  Production  Tima 
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. Figure  8b.  Factor  Affecting  Time  to  Complete  Workload 


this  in  mind  we  shall  define  the  term 
ability"  to  represent  the  frequency  of 
malfunctions,  and  "Maintainability"  to  repre¬ 
sent  the  amount  of  time  when  the  system  has 
reduced  capability  due  to  required  repairs  or 
replacement.  Thus  the  system  designer  must 
make  certain  that  the  net  system  capability 
will  enable  the  workload  to  be  processed  in 
the  required  time. 

These  failure  and  down  time  considerations 
are  illustrated  in  Figure  9.  System  Avail¬ 
ability  is  defined  as  the  proportion  of  Up 
Time  to  Total  Time.  The  set  of  these  factors 
may  be  treated  in  the  following  way: 

a.  Make  certain  that  the  total  down 
time  and  associated  reduction  in  system 
capability  is  taken  into  account  when  design¬ 
ing  the  system  to  meet  the  required  workload. 

b.  Frequency  of  failure  or  system 
availability  may  also  be  treated  as  a  system 
constraint,  i.e.,  the  maximum  frequency  of 
failure  that  can  be  tolerated. 

c.  All  of  the  maintenance  factors 
finally  result  in  added  cost,  and  will  be 
accounted  for  in  the  Present  Value  Life  Cycle 
Cost  (PVLCC)  analysis  of  each  system. 

9.3  Dealing  With  an  Uncertain  Workload 

The  previous  analysis  of  system  capa¬ 
bility  was  based  on  the  assumption  that  the 
input  workload  was  known  exactly,  as  illus¬ 
trated  in  Figure  8a.  Sometimes  the  assump- 
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Cion  is  made  Chat  this  is  Che  minimum  manda- 
Cory  workload  but  ChaC  ezCra  credic  will  be 
given  for  syscems  having  Che  capabilicy 
greaCer  Chan  Che  minimum.  The  designer's 
problem  is,  how  much  extra  capabilicy  is 
desired  over  cime?  Many  Cimes  Che  clienC 
cannoC  accuracely  predicC  what  Che  acCual 
workload  may  be.  However,  some  limics  muse 
be  sec  if  appropriaCe  guidance  is  Co  be  given 
co  Che  designers. 

One  way  of  dealing  wich  Chis  uncerCainCy  is 
Co  express  Che  workload  as  a  probabiliCy 
discribucion  as  shown  in  Figure  10.  SeCCing 
Che  upper  limic  is  fairly  scraighcforward, 
since  Chis  can  be  sec  as  an  arbiCrary  design 
limic  beyond  which  addicional  capabilicy  is 
assumed  Co  have  no  value.  InCermediaCe 
probabiliCy  values  can  Chen  be  inserCed,  as 
shown  in  Figure  10,  using  whaCever  daCa  Che 
clienC  has  available  (richer  sCadscical  daCa 
or  judgmencal  estimaCes) . 

Based  on  Chis  assumed  workload,  Che  designer 
muse  Chen  design  Che  system  Co  be  able  to 
meet  the  entire  range  of  workload  levels, 
over  cime,  adding  addicional  system  incre¬ 
ments  whenever  required.  In  Che  illustration 
of  Figure  10,  the  designer  proposed  System 
as  the  initial  system.  This  system  will 
"absolutely"  meet  the  workload  requirement  in 
Years  1  and  2  and  will  "absolutely  not"  meet 
the  requirement  in  Year  5.  However,  the 
designer  proposed  Co  add  an  addicional 

capabilicy  to  A^,  to  yield  System  A2, 
whenever  A2  is  needed.  Generally  two 

constraints  are  placed  on  the  designer  for 
purposes  of  design  and  evaluation: 
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o  No  more  than  one  or  two  growth  addi¬ 
tions  will  be  permitted  to  keep 
disruptions  within  acceptable  limits. 

o  An  addition  will  be  required  whenever 
the  operational  hours  per  month 
reaches  some  upper  limit  (say,  600 
hours  per  month  as  shown  in  Figure 
10).  This  will  permit  sufficient 
time  for  corrective  and  preventive 
maintenance.  Alternatively,  this 
limit  could  be  a  function  of  the 
system's  demonstrated  maintainability. 

Providing  such  system  "upgrades"  during  its 
operational  life  saves  the  user  money  since 
it  prevents  the  user  from  having  to  buy  more 
capability  than  is  required  (like  A2)  at 
the  beginning  of  system  operations  rather 
than  when  it  is  actually  needed. 

Finally  the  designer  should  provide  "credible 
evidence"  for  the  client  that  the  preferred 
system  can  in  fact  meet  the  entire  defined 
workload  (as  shown  in  Figure  10)  over  its 
entire  levels,  as  well  as  all  constraints. 
For  off-the-shelf  systems,  this  could  be 
validated  by  live  test  demonstrations.  F„r 
development  type  projects  chis  could  be  shown 
by  simulation  or  analysis. 

9. A  Availability  Dates  and  Schedule  Risk 

Given  that  each  system  has  been  designed 
to  provide  the  required  net  productivity  to 
meet  the  incoming  workload  specified  in 
Figures  8a  or  10,  ve  shall  now  describe  how 
the  designer  plans  the  development  effort  to 
meet  the  required  availability  dates  (t^ 
and  t2)  and  provides  the  necessary  data  to 
the  client,  enabling  him  to  perform  his 
validation  function.  Recall  that  Figure  A, 
which  displays  the  project  work  process  in 
network  form  and  the  time  estimates  of  the 
activities  along  the  critical  path,  was  used 
to  calculate  the  project  completion  date  as 
the  normal  distribution  of  Figure  5c.  In  the 
example  shown,  there  is  approximately  a  A0Z 
chance  of  successfully  completing  the  project 
by  date  Tj  (60Z  chance  of  schedule  over¬ 
run).  However,  the  client  may  not  be 
comfortable  with  this  degree  of  risk.  Thus, 
the  system  designer  must  find  out  what  risk 
of  overrun  the  client  is  willing  to  accept. 
Assume  a  10Z  risk  is  acceptable.  As  shown  in 
Figure  5c,  for  this  10Z  risk  the  project 
would  be  completed  on  or  before  T2  which  is 
later  than  T^  and  hence  unacceptable. 
Thus,  the  system  designer  must  reconfigure 
the  project  plan  to  reduce  completion  date. 
Generally  this  is  done  by  adding  more  re¬ 
sources  on  one  or  more  activities  on  the 
critical  prth  until  the  new  completion  date 
probability  function  satisfying  the  require¬ 
ment  (10Z  chance  of  overrun  at  time  Tj)  is 
obtained,  as  shown  in  Figure  5c. 


Figure  10.  Designer  Response  to  Meeting 
Workloea  rrobebilities 


In  summary,  Schedule  Risk  is  defined  as  the 
probability  that  the  project  will  overrun  a 
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required  delivery  date.  For  purposes  of  Che 
p leaning  end  eveluetion  effort*,  the  velue  of 
Schedule  Risk  eccepteble  to  the  client  should 
be  provided  to  the  designer  et  the  beginning 
of  the  study.  Then  it  is  up  to  the  designer 
to  construct  the  project  work  plsn  accord¬ 
ingly  and  provide  the  project  activitiea 
network  plus  all  time  calculations  to  show 
that  the  completion  date,  as  a  probability 
function,  satisfies  both  the  time  and  riak 
requirements. 


9.5  Consideration  of  Total  Cost 

The  project  activitie's  network  (Fig¬ 
ure  4)  is  also  the  starting  point  for  the 
cost  calculations.  From  this  network  the 
cost  of  all  work  and  cost  elements  (EDT&E, 
procurement,  installation,  operations, 
maintenance,  and  support)  borne  by  the  client 


must  be  calculated  for  each  year 


1 


Here 


colt  *t*nd*rd  f<?rrla8  calculating  the 
cost  of  an  activity  are  used  (e.g.,  labor 

IZ  equ‘1*  th'  product  of  man-hours  and 
labor  rafe).  When  the  activity  time  is 
expressed  a.  .  three-point  estimate,  these 
tme.  suit  be  converted  to  an  expected  value 
*nd  a  standard  deviation  (as  shown  previously 
in  Figure  5a).  These  value,  for  time  are 
then  converted  to  labor  cost  value,  by 
multiplying  each  by  the  labor  rate.  Costs 
ere  then  accumulated  by  year,  and  the 
expected  value  of  cost  and  the  cost  variance 
for  each  year  are  calculated  as  follows:  The 
expected  total  cost  for  the  year  is  equal  to 
the  sum  of  the  expected  costs.  The  total 
cost  variance  for  each  year  ia  equal  to  the 
sum  of  the  squares  of  the  individual  standard 
deviations  for  that  year. 


hv.:  When  a  probabilistic  workload  is  assumed  (as 

discussed  in  the  previous  section),  each 
yearly  cost  must  be  handled  as  a  frequency 
distribution.  Operations,  maintenance,  or 
leasing  costs  must  be  calculated  for  each 
■.!&$  probability  segment.  Using  Figure  10  as  an 
'■'f-  example,  first  calculate  the  cost  of  segment 
Si  as  a  function  of  its  operating  time. 
Using  the  time  of  the  mid-point  of  the  pi 
segment,  and  using  the  pricing  data  supplied 
the  designer  ss  well  as  the  user  labor  cost, 
calculate  the  cost  associated  vith  this 
mid-point  time.  This  cost  has  the  probabil¬ 
ity  p|  “  .20  associated  with  it.  Make  a 
similar  calculation  for  the  other  segments, 
®2»  ®3»  ,nd  S4.  Then  calculate  the 

expected  value  and  standard  deviation  for 
these  four  probabilities.  These  terms  are 
then  added  as  part  of  the  sum  of  the  other 
expected  values  and  the  squares  of  the 


*In  this  section  we  assume  that  the  cost  of 
all  work  performed  by  the  client  and  the  user 
is  included  in  the  analysis.  If  other 
organisations  or  the  public  glso  perform  work 
or  bear  any  of  the  costs,  such  costs  must 
»l»o  be  factored  into  the  analysis. 


standard  deviations  of  the  cost  terms  of  that 
yea?  as  described  previously. 

The  expected  value  and  standard  deviation  of 
cost  for  each  year  must  then  be  incorporated 
into  a  present  value  analysis  in  the  follow¬ 
ing  way.  First,  apply  the  appropriate 
discount  rate  to  each  of  the  yearly  expected 
values  of  cost.  The  sum  of  this  result  is 
the  present  value  of  expected  cost.  The 
present  value  standard  deviation  is  calcu¬ 
lated  as  follows: 

o  Find  the  standard  deviation  of  cost 
for  each  year  as  the  square  root  of 
the  variance  of  cost  for  each  year. 

o  Apply  the  appropriate  discount  factor 
to  each  year's  standard  deviation. 
This  yields  the  present  value  of  each 
year's  standard  deviation. 

o  Square  each  of  these  factors  to 
obtain  the  present  value  of  each 
year's  variance. 

o  Sum  each  of  these  present  value  vari¬ 
ances. 

o  Take  the  square  root  of  this  sum. 
This  is  the  standard  deviation  of 
total  cost. 


A  lease-versus-buy  calculation  can  also  be 
made  by  including  the  purchase  cost  of  the 
system.  But  this  must  also  be  handled  on  a 
probabilistic  basis.  In  the  example  of  Fig¬ 
ure  10,  there  is  a  100Z  chance  that  System 
Ai  will  be  required  for  the  initial 
installation.  As  seen  in  Figure  10,  there  is 
s  56Z  chance  that  System  A2  will  be  needed 
in  Year  3,  85Z  chance  in  Year  4,  and  100Z 
chance  in  Year  5.  Using  this  data,  the 
probability  of  purchasing  A2  is  each  year 
(given  that  it  was  not  purchased  previously) 
can  be  calculated.  This  data  can  be 
converted  into  an  expected  value  and  standard 
deviation  of  purchase  price  for  that  year  and 
these  values  added  to  the  other  yearly 
expected  values  and  standard  deviations  as 
described  previously. 

Having  calculated  the  present  value  of  the 
expected  value  and  standard  deviation  of 
cost,  the  PVLCC  can  be  obtained  as  a  normal 
distribution,  similar  to  Figures  5b  and  5c. 
Now  the  factor  of  cost  risk  can  be  introduced 
in  the  same  way  as  schedule  risk  was  treated. 
Namely,  the  client  should  indicate  the  cost 
risk  they  are  willing  to  assume,  where  cost 
risk  is  defined  as  the  probability  of  cost 
overrun.  "or  example,  the  expected  value  of 
cost  has  a  50Z  chance  of  overrun  and  this  may 
be  unsatisfactory  to  the  client.  If  the 

client  is  only  willing  to  assume  a  10Z  chance 
of  overrun,  for  example,  this  amount  is 
applied  to  the  cost  probabilistic  values,  as 
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illustrated  in  Figure  5c,  and  the  value  of 
cost  obtained.  For  this  example  of  a  10Z 
chance  of  overrun,  the  value  of  cost  “  Ce  + 
1.28. 

where  Ce  “  expected  value  of  cost 

< Tm  standard  deviation  of  cost 

Values  of  cost  for  other  values  of  risk  may 
be  similarly  calculated  using  data  from  a 
standard  normal  probability  distribution. 

9.6  Consideration  of  Growth  Factors 

Here  are  three  other  related  factors 
which  often  arise  in  an  evaluation: 

o  Upward  Compatibility 

o  Growth  Potential 


same  one  used  in  the  case  of  job  uncertain¬ 
ties  (Figure  10).  That  is,  the  client  will 
validate  that  the  total  system,  including 
changes,  is  capable  of  handling  the  total 
workload,  with  proper  response  times,  if  it 
should  occur.  In  addition,  the  client's 
evaluator  will  calculate  the  total  cost  on  a 
probabilistic  basis  and  apply  the  cost  risk 
factor  to  estimate  the  total  cost  to  be  used 
in  the  evaluation. 


9.7  Consideration  of  Superior  Characteristics 
The  recommended  evaluation  approach 
described  thus  far  can  be  sunmarixed  as 
follows : 


o  All  systems  have  been  designed  to 
perform  the  same  set  of  operational 
jobs  and  to  meet  all  specified 
constraints. 


o  Flexibility 

Here  is  a  description  of  how  they  would  be 
treated  under  this  evaluation  method. 

The  first  step  is  to  understand  and  define 
what  the  client  means  by  these  terms.  Gen¬ 
erally  the  term  "Upward  Compatibility"  is 
used  to  connote  that  the  system  can  be 
reconfigured  to  provide  greater  capability  by 
adding  additional  elements.  That  is,  it  can 
be  modified  by  using  all  or  part  of  the 
original  system,  thus  providing  the  greater 
capability  at  less  cost  and  disruption  than 
if  a  second,  totally  new  system  were  used. 

"Growth  Potential"  is  quite  related  to  the 
previous  definition  of  Upward  Compatibility. 
In  this  case  the  size  of  the  job  may  be 
"growing"  or  increasing,  and  hence  a  larger 
capability  may  be  needed. 

"Flexibility"  generally  means  that  the  set  of 
jobs  may  change,  and  the  client  would  like 
the  original  system  to  be  sufficiently 
general-purpose  so  that  its  capabilities  are 
sufficient  to  perform  the  new  set  of  jobs 
rather  than  just  the  original  set  of  jobs. 

Thus,  all  three  of  these  terms  suggest  that 
the  client  has  some  other  set  of  jobs  in  mind 
besides  the  originally  defined  set  of  jobs. 
In  keeping  with  the  evaluation  approach 
described,  here  is  how  these  terms  may  be 
included  in  the  evaluation.  First,  ex¬ 
plicitly  define  a  representative  set  of  other 
jobs  which  may  be  required  to  be  performed  by 
the  system.  Second,  indicate  both  the  dates 
when  these  jobs  will  be  performed  (such  as  in 
Tears  4  and  5)  and  the  probability  that  the 
jobs  will  occur.  This  may  be  a  subjective 
estimate  of  the  probabilities.  Thus  both 
sets  of  workloads  now  become  the  total 
requirement.  And  the  designer  is  required  to 
configure  h^a  system  design  to  accommodate 
the  total  set  of  jobs.  In  general,  the 
design  and  evaluation  approach  used  is  the 


o  All  systems  will  become  available  at 
the  specified  date(s),  taking  into 
account  an  acceptable  risk  of  sched¬ 
ule  overrun. 

o  The  preferred  system  is  the  one  which 
requires  the  lowest  cost  (PVLCC), 
taking  into  account  an  acceptable 
risk  of  cost  overrun. 

However,  sometimes  a  designer  provides 
one  or  more  characteristics  (generally  at 
greater  cost)  which  are  clearly  superior  to 
the  lowest  cost  system  alternative.  Nov  the 
question  raised  is,  are  these  incremental 
superiorities  provided  worth  the  difference 
in  cost? 

The  key  factor  to  be  analyzed  is,  have 
these  superior  characteristics  been  con¬ 
sidered  in  performing  the  operational  jobs 
which  have  been  evaluated?  Or  are  there 
other  jobs  which  would  demonstrate  each  of 
the  superior  system's  characteristics?  In 
the  former  case,  a  system's  superior  char¬ 
acteristics  may  have  already  been  accounted 
for  in  the  cost  calculations.  Hence,  no 
further  "credits"  need  be  given  to  that 
system.  In  the  latter  case,  the  evaluator 
can  calculate  the  additional  "credits"  to  be 
given  as  follows: 

o  Clearly  define  all  other  jobs  to 
which  these  superior  characteristics 
would  apply. 

o  Estimate  how  much  additional  cost 
would  have  to  be  paid  by  the  client 
if  the  lowest  cost  system  were  used 
for  these  jobs  rather  than  the 
superior  system.  This  cost  is  obvi¬ 
ously  a  function  of  how  often  each 
job  is  performed  during  the  system 
life  cycle,  or  the  probability  of  its 
being  performed.  This  additional 
cost  should  be  added  to  the  lowest 
cost  system  to  determine  what  the 
true  PVLCC  would  be  for  all  systems. 


242 


Note  that  what  we  have  done  is  to 
enlarge  the  set  of  operational  jobs  to  be 
done,  and  enlarged  the  total  costs  required 
to  do  them.  Thus  this  new  total,  cost  can  be 
the  basis  of  the  system  selection. 

PARI  IV.  APPLYING  EVALUATION  METHOD  TO  A 
PROPOSAL 

The  previous ' sections  presented  a  method  for 
performing  an  architectural  design  study 
(involving  systems'  analysis,  synthesis  and 
~ri  evaluation  of  alternatives)  in  an  environment 
where  there  is  close  contact  with  the  cli¬ 
ent.  A  proposal  effort  fundamentally  in¬ 
volves  the  same  systems  planning  functions  as 
described  for  the  architectural  study.  How¬ 
ever,  instead  of  the  designer  synthesizing 
and  evaluating  all  of  the  system  alternatives 
and  selecting  the  preferred  one,  a  set  of 
competing  designers  each  designs  a  proposed 
system  and  submits  these  to  the  client 
evaluators  for  their  selection.  Here  are  the 
differences  which  make  it  more  difficult  to 
"optimize"  a  system  in  a  proposal  effort  than 
.  ‘  in  an  architectural  design  study: 

o  Pirst,  the  system  requirements  are 
generally  in  the  form  of  technical 
specifications  with  firm  mandatory 
requirements.  This  may  force  the 
designer  to  provide  extra  capabili¬ 
ties  if  the  off-the-shelf  entities  to 
be  employed  do  not  exactly  match  the 
mandatory  requirements. 

o  The  second  and  most  important  dif¬ 
ference  is  that  there  generally  is 
little  opportunity  to  make  contact 
with  the  client  prior  to  submission 
of  the  proposal,  and  hence  it  is  more 
difficult  to  "optimize"  the  design  in 
terms  of  the  client's  desires.  Thus 
it  is  very  important  that  the  de¬ 
signer  review  the  Request  for  Pro¬ 
posal  and  make  certain  that  he  under¬ 
stands  what  the  client  is  requesting 
and  the  details  of  the  evaluation 
method  to  be  used.  There  should  be 
an  opportunity  for  the  designer  to 
obtain  clarification  of  any  fact 
which  is  ambiguous  to  him. 

With  these  differences  in  mind,  we  shall  now 
describe  how  to  apply  the  previous  systems 
planning  approach  to  the  client  proposal 
process. 

10.  APPLICATION  OP  METHOD  TO  A  TECHNICAL 
CHARACTERISTICS  TYPE  OF  PROCUREMENT 

£  In  this  scenario  it  is  assumed  that  the 

client  provides  the  system  requirements 
primarily  in  the  form  of  technical  specifica¬ 
tions  rather  than  operationally  oriented 
jobs.  The  system  design  and  evaluation 
approach  nov  recommended  will  still  be  based 


; 


y 
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on  the  approach  previously  presented  but  with 
the  following  changes  as  indicated. 

o  Technical  specifications  are  again 
presented  as  two  levels:  1)  a 

mandatory  minimum,  and  2)  desirable 
features. 

o  An  additional  aid  to  the  designers 
would  be  the  inclusion  of  operation¬ 
ally  oriented  information  regarding 
the  operational  use  of  the  system 
(jobs  and  functions  to  be  performed). 

o  All  desirable  features  will  be  de¬ 
scribed,  and  the  value  of  providing 
each  of  these  features  will  be  pro¬ 
vided  to  all  designers.  These  values 
will  be  derived  from  the  architec¬ 
tural  design  studies  which  were  per¬ 
formed  at  some  previous  time,  and 
which  were  the  basis  of  the  technical 
specifications.  Based  on  the  archi¬ 
tectural  studies,  the  client  should 
also  provide  the  designers  with  eval¬ 
uation  functions  indicating  the  worth 
of  exceeding  the  mandatory  minimum 
requirements.  That  is,  what  is  the 
value  of  exceeding  a  minimum  manda¬ 
tory  requirement  in  terms  of  its 
dollar  savings  somewhere  else.  As 
described  previously,  each  of  these 
values  is  equal  to  the  lowest  addi¬ 
tional  cost  of  performing  the  jobs 
needing  these  functions  (or  providing 
additional  performance)  if  the  func¬ 
tions  (or  additional  performance) 
were  not  provided. 

o  Each  designer  would  then  attempt  to 
design  a  system  exactly  meeting  each 
mandatory  requirement.  However,  the 
designer  will  also  consider  if  it  is 
possible  to  make  trade-offs  among 
related  parameters  which  will  meet  a 
joint  requirement  at  lower  cost  than 
the  cost  of  meeting  (or  exceeding) 
the  requirements  singularly,  taking 
into  account  the  value  of  the  addi¬ 
tional  features  or  performance. 

o  Ideally,  the  client  would  provide  the 
designer  with  the  value  of  exceeding 
the  mandatory  requirements.  Each 
designer  could  then  properly  "opti¬ 
mize"  his  proposal  in  terms  of  meet¬ 
ing  all  requirements  at  lowest  PVLCC 
to  the  client,  taking  into  account 
the  value  of  desirable  features  as 
well  as  all  significantly  superior 
characteristics.  However,  if  the 
client  does  not  provide  these  values 
and  the  designer  finds  he  must 
include  these  "extras"  in  his  design, 
he  should  estimate  its  value  using 
the  method  described  previously. 

o  In  either  case  the  client  should  also 
validate  such  calculations  and  select 
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that  designer  which  meets  all  re¬ 
quirements  and  performs  all  jobs  at 
lowest  total  PVLCC. 

o  Developmental  uncertainties  as  re¬ 
flected  into  schedule  risk  and  cost 
risk  would  be  treated  in  the  same  way 
as  previously  described. 

PAST  V.  CONCLUSIONS 

In  this  paper  we  have  presented  several 
potential  pitfalls  which  can  occur  in  the 
process  o'f  designing,  evaluating  and  select¬ 
ing  the  preferred  system  for  clients.  Some 
are  fairly  obvious;  some  are  not.  These 
pitfalls  and  other  conclusions  reached  in 
this  paper  can  be  susmarized  as  follows: 

a.  Unless  the  requirements  of  the  job 

are  clearly  stated  and  understood  by  the 

designers,  they  will  not  be  able  to  design 
their  systems  appropriately.  Thus,  some 
means  should  always  be  available  for  further 
discussion  and  clarification  of  these  re¬ 

quirements  prior  to  the  start  of  system 
design  efforts.  This  opportunity  is  gen¬ 
erally  available  to  designers,  and  should  be 
utilized  early  in  the  design  process. 

b.  Unless  an  objective  procedure  for 

evaluating  system  alternatives  is  provided  to 
the  designers  by  the  client,  designers  will 
not  be  able  to  perform  their  cost-performance 
trade-offs  effectively  to  arrive  at  the 
system  design  preferred  by  the  client. 


c.  The  system  requirements  should  be 

stated  in  a  way  that  will  enable  the  designer 
to  provide  what  is  desired  by  the  client  at 
lowest  total  cost.  In  architectural  studies 
where  the  client  desires  the  designer  to  make 
systems  engineering  trade-offs  among  the  key 
design  parameters,  it  is  preferable  that  the 
requirements  should  be  stated  as  operational 
jobs  to  be  done  rather  than  a  set  of  detailed 
system  characteristics.  If  the  client  also 
wishes  to  include  a  set  of  technical  char¬ 
acteristics  as  mandatory  minimum  require¬ 
ments,  with  additional  desirable  features,  it 
would  be  helpful  to  list  each  design  con¬ 
straint  in  two  ways:  1)  a  design  goal, 

indicating  the  client's  mandatory  minimum 
value  which  must  be  equalled  or  surpassed, 
and  2)  the  worth  of  exceeding  this  minimum 
value.  By  doing  this  the  designer  should  be 
permitted  to  make  appropriate  trade-offs 
among  design  parameters  and  thus  be  better 
able  to  satisfy  the  user  needs  at  lowest 
cost.  The  same  approach  should  be  used  in 
proposal  efforts. 

d.  Credible  evidence  of  the  accuracy 
and  reliability  of  the  proposed  work  plan 
should  be  provided  to  the  client  as  part  of 
the  architectural  design  study  and  proposal 
efforts.  Such  evidence  includes:  1)  per¬ 
formance  validation  through  live  test  demon¬ 
strations,  simulations  or  analysis;  2) 
reliability,  maintainability  data,  when 
available;  3)  schedule  analysis,  including 
critical  path  analyses;  4)  cost  analyses;  and 
5)  risk  analyses. 


RISK  ANALYSIS  TRAINING  WITHIN  THE  ARMY: 
CURRENT  STATUS,  FUTURE  TRENDS 


by  Joseph  G.  King 

US  Army  Logistics  Management  Center 


The  analysis  of  technical  risk  was  to 
have-  become  a' way  of  life  for  Army  program 
managers  back  in  1970  when  the  US  Army  Materi¬ 
el  Development  and  Readiness  Conmand  (DARCOM) 

-•  established  a  program  called  "Program  for  the 
Refinement  of  the  Materiel  Acquisition  Pro¬ 
cess"  (PR0MAP-70).  One  task  of  PR0MAP-70  was 
to  improve  the  analysis  of  technical  risk  so 
that  cost  growth  did  not  result  from  prolonged 
development  or  from  changes  required  to  over¬ 
come  technical  problems.  However,  13  years 
later,  risk  analysis  is  still  not  commonplace. 
Some  systems  are  fielded  without  risk  being 
formally  addressed.  Articles  are  still  being 
written  about  DOD  waste  and  mismanagement  of 
acquisition  programs.  An  Army  course  in  ad¬ 
vanced  decision  risk  analysis  was  developed 
and  then  canceled.  CARCOM  even  considered 
deleting  the  regulation  that  requires  the  per¬ 
formance  of  a  risk  analysis  for  major  system 
acquisitions. 


What  has  happened  since  1970?  Has  DOD 
directed  that  formal  risk  analyses  are  a  waste 
of  time  and  are  not  to  be  performed?  No.  In 
1969,  Deputy  Secretary  of  Defense  David 
Packard  sent  a  memorandum  to  all  military  de¬ 
partments  highlighting  problems  in  major  sys¬ 
tem  acquisition.  This  memorandum  was  what 
prompted  the  Army  to  develop  PR0MAP-70.  In 
1981,  Deputy  Secretary  of  Defense  Frank 
Carlucci,  III,  published  a  memo  titled  "Im¬ 
proving  the  Acquisition  Process."  One  of  the 
"initiatives"  requires  DOD  action  to  increase 
the  visibility  of  technical  risk  in  budgets  of 
weapon  system  acquisition  programs. 


What  is  the  problem?  Should  there  be 
more  regulations?  In  DARCOM,  the  existing 
regulation  is  often  ignored  and,  where  it  is 
followed,  the  people  doing  the  analysis  wonder 
if  the  study  has  any  bearing  on  decisions  made. 
Is  the  study  being  done  just  because  the  reg¬ 
ulation  says  to  do  it?  /  .  > 


Specialized  training  may  be  an  answer. 

"^The  purpose  of  this  paper  is  to  outline  what 
is  done  by  the  Army  to  train  people  in  risk 
analysis.  Since  risk  analysis  training  In  the 
Army  is  provided  by  the  Army  Logistics  Manage¬ 
ment  Center  (ALMC),  my  focus  is  on  the  courses 
offered  there.  After  providing  a  brief  his¬ 
tory  of  how  ALMC  became  involved  in  risk  anal¬ 
ysis  training,  ^reviev^all  risk  analysis- 
related  training  'done  in  any  course  at  the 
Center.  I-prorrde-  information  on  the  student 
population  of  each  fourse,  what  risk  subjects 
are  taught,  approximately  how  much  time  is^.-/ 
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spent  on  the  subject,  and  how  the  students  are 
expected  to  use  the  training.  Then  the  future 
r  trend  in  risk  analysis  training  is  presented. 

I •  oirtVi rre- new  courses,  course  changes  and 
hardware/software  changes  that  will  make  risk 
analysis  more  palatable.  ' . -  j-  “ 


ALMC  has  been  involved  in  risk  analysis 
instruction  since  1970.  The  Center  was  tasked 
with  developing  a  capability  to  train  students 
to  analyze  technical  risk  as  part  of  PR0MAP-70. 
ALMC  was  not  prepared  to  meet  this  task  with¬ 
out  impacting  adversely  on  other  courses,  so 
the  instruction  was  contracted  out  to  Mathe- 
matica,  Inc.,  of  Princton,  NJ.  Mathematica 
put  together  a  one-week  course  called  Intro¬ 
duction  to  Risk  Analysis.  They  also  developed 
a  simulation  model  called  MATHNET  which 
evolved  into  models  such  as  SOLVENET,  RISCA, 
and  VERT.  Within  a  year,  ALMC's  staff  took 
over  the  instruction  of  the  course.  Workshops 
were  added  to  the  course  and  the  course  length 
was  extended  by  a  week,  giving  a  format  simi¬ 
lar  to  what  is  presented  today  in  the  Decision 
Risk  Analysis  Course  (DRAC). 

A  second  course  in  risk  analysis  was  also 
developed  in  March  1973.  ALMC  was  urged  by 
DARCOM  to  initiate  a  second  course  in  risk 
analysis  geared  to  logisticians.  This  course 
developed  into  Decision  Risk  Analysis  for 
Logisticians  (DRALOG)  which  is  still  taught 
today.  An  Advanced  Decision  Risk  Analysis 
Course  was  also  developed.  This  course  was  to 
go  beyond  the  basics  as  taught  in  the  other 
courses.  It  was  canceled  in  1982  due  to  low 
class  fill.  ALMC  also  included  some  risk  in¬ 
struction  in  five  other  classes. 

PRESENT  INSTRUCTION 


Today,  there  are  still  five  courses  in 
which  risk  analysis  is  taught.  Project  Manage¬ 
ment  Development  Course  (PMDC)  is  a  5-week 
course  of  which  8  to  10  hours  are  devoted  to 
risk  analysis.  The  student  receives  a  brief 
history  of  formal  risk  analysis  in  the  Army. 
They  discuss  the  reason  for  risk  analysis  and 
the  process  called  Decision  Risk  Analysis. 

They  are  also  exposed  to  one  of  the  most  ad¬ 
vanced  risk  analysis  tools  called  Venture  Eval¬ 
uation  Review  Techniques  (VERT).  These  stu¬ 
dents  return  to  their  project  management  shops 
informed  about  the  basics  of  risk  analysis.  If 
they  desire  to  use  the  tools  mentioned  in 
class,  they  have  to  send  people  to  be  trained 
further  in  the  risk  analysis  techniques. 

ORSA  MAC  I  is  a  12-week  course  of  which 
approximately  56  hours  are  devoted  to  risk 
analysis-related  subjects.  Fifty  hours  are 
spent  on  decision  trees,  multiattribute  deci¬ 
sion  analysis,  utility  theory,  and  6  hours  are 
spent  on  VERT.  The  graduates  of  this  course  are 
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mostly  military  who  are  enroute  to  an  SC  49 
(ORSA)  assignment.  Because  they  are  ORSA- 
trained,  they  are  expected  to  know  a  myriad 
of  tools  and  techniques.  Risk  analysis  Is  one 
of  many  tools  with  which  they  are  equipped. 

ORSA  MAC  II  is  a  3-week  course  with  about 
15  ,  iurs  of  risk  instruction.  The  bulk  of  the 
time  is  spent  on  decision  analysis  techniques 
with  3  hours  devoted  to  VERT.  The  student  in 
this  class  comes  from  the  same  population  as 
those  who  attend  ORSA  MAC  I;  however,  the  MAC 
II  students  have  prior  ORSA  training.  Since 
they  all  come  to  the  class  after  working  in  a 
non-ORSA  job,  this  course  is  used  to  refresh 
their  ORSA  skills  prior  to  returning  to  an 
ORSA  assignment. 

This  brings  me  to  the  only  two  classes 
offered  in  the  Army  (and  possibly  DOD)  which 
are  entirely  devoted  to  decision  and  risk 
analysis--DRAC  and  DRALOG.  Decision  Risk  An¬ 
alysis  for  Logisticians  is  a  2-week  course 
geared  to  reach  an  audience  who  have  a  non- 
quantitative  background.  These  students' 
duties  involve  them  in  all  aspects  of  logis¬ 
tics  management.  The  purpose  of  the  class  is 
to  give  the  student  a  detailed  introduction  to 
the  concepts  of  risk  analysis  so  that  they  can 
use  risk  analysis  in  the  logistics  or  readi¬ 
ness  areas.  Graduates  are  capable  of  Initiat¬ 
ing  action  with  their  systems  analysis  people 
to  do  a  risk  analysis,  or  participating  in 
risk  analysis  studies  that  require  their  ex¬ 
pertise.  To  accomplish  this,  the  students 
spend  4  hours  discussing  why  risk  analysis  is 
important  and  how  it  developed  within  the 
Army.  They  are  exposed  to  12  hours  of  quanti¬ 
tative  techniques  used  in  risk  analysis  start¬ 
ing  with  SET  theory  and  moving  through  proba¬ 
bility  distributions.  They  are  taught  8  hours 
of  decision  analysis  techniques  to  Include  de¬ 
cision  trees,  simple  ranking  techniques,  and 
utility  theory.  Another  4-hour  block  of  in¬ 
struction  is  used  to  explain  risk  analysis  In 
general  and  to  introduce  the  student  to  con¬ 
cepts  modeling  and  Monte  Carlo  slmualtion. 
Eight  hours  are  devoted  to  a  specific  Monte 
Carlo  simulation  technique  called  RISCA  III. 
The  remaining  4  hours  of  the  first  week  are 
used  to  address  how  subjective  estimates  are 
used  in  the  analysis.  The  second  week  of  the 
course  is  devoted  to  workshops.  The  class  is 
presented  a  case  each  day  to  work  on  where 
they  are  to  analyze  the  risk  and  choose  the 
"best"  alternative  after  the  risk  for  each 
alternative  has  been  developed.  The  last  two 
cases  in  this  course  are  structured  to  show 
that  risk  analysis  is  not  just  for  RSD  ven¬ 
tures,  but  is  also  useful  to  analyze  policy  In 
the  readiness  arena. 

The  Decision  Risk  Analysis  Course  is 
also  a  2-week  course  but  it  is  designed  for 
OR  Analysts,  Engineers  and  Scientists.  The 
majority  of  the  subject  areas  taught  in  the 


first  week  of  DRAC  are  similar  to  DRALOG.  The 
major  differences  between  the  courses  are  the 
pace  and  the  amount  of  material  covered.  DRAC 
goes  into  more  detail  in  the  same  amount  of 
time  used  in  DRALOG,  so  the  pace  is  faster  in 
DRAC.  Another  difference  is  the  simulation 
model  taught.  In  DRAC  the  student  learns  VERT 
instead  of  RISCA  III  as  in  DRALOG.  The  second 
week  of  this  class  is  devoted  to  workshops  as 
with  DRALOG;  however,  all  of  the  cases  in 
DRAC  are  R&D,  test  and  evaluation  oriented. 

By  the  end  of  the  class,  these  students  become 
quite  proficient  at  using  VERT  to  analyze 
risk.  This  class  is  a  must  for  anyone  in  the 
Army  who  plans  on  doing  risk  analysis. 

FUTURE  TRENDS 

The  future  of  risk  analysis  training  hold 
a  number  of  changes.  The  changes  are  due  to  a 
study  that  was  conducted  by  Dr.  Erwin  Atzinger. 
Chief,  Special  Studies  and  Activity  at  AMSAa. 

In  1982,  Dr.  Atzinger  studied  the  use  of  sys¬ 
tems  analysis  techniques  at  the  DARCOM  sub¬ 
ordinate  commands.  As  part  of  this  study,  he 
looked  at  the  use  of  Decision  Risk  Analysis 
and  the  perception  of  the  training  being  of¬ 
fered  in  this  area.  The  results  showed  that 
the  training  was  well  received,  but  it  is  not 
enough  to  overcome  the  resistance  that  exists 
against  the  use  of  the  tool.  Also,  there  was 
some  conment  the'  there  is  not  enough  taught 
on  how  to  do  a  qualitative  risk  analysis. 

Some  felt  ther^  was  too  much  emphasis  on  VERT, 
while  others  would  like  to  see  more  VERT. 

Since  Dr.  Atzinger  is  also  the  proponent  for 
the  risk  courses  at  ALMC,  he  passed  the  com¬ 
ment  on  to  us  so  the  changes  could  be  evalu¬ 
ated.  As  a  result,  the  following  actions  are 
planned  by  ALMC. 

ALMC  will  offer  a  2-  to  4 -hour  briefing 
targeted  toward  the  highest  management  people 
within  the  Army  and  possibly  DOD.  The  purpose 
will  be  to  explain  to  the  chiefs  of  each  com¬ 
mand  what  tools  are  available  to  them  in  the 
risk  analysis  area  and  how  these  tools  can 
benefit  them.  It  will  be  tailored  to  be  of¬ 
fered  to  commanders  and  their  staffs  or  at 
conferences  such  as  the  Project  Managers’  con¬ 
ference.  This  briefing  should  be  available 
early  in  FY  84. 

DRAC  will  be  modified  by  eliminating  SET 
theory  and  some  of  the  VERT  workshops  to  make 
room  for  more  qualitative  risk  analysis  in¬ 
struction  and  more  subjective  estimation.  SET 
theory  will  be  put  into  a  progranmed  text  that 
the  student  will  be  expected  to  review  prior  tc 
coming  to  ALMC.  The  VERT  workshops  will  have 
to  give  way  to  qualitative  risk  analysis  work¬ 
shops  and  "decision  analysis  workshops.  Shift¬ 
ing  from  VERT  in  DRAC  is  due  to  the  perceived 
overemphasis  now  in  the  present  course.  Sixty 
percent  of  the  present  DRAC  is  either  VERT  in¬ 
struction  or  VERT  workshops.  When  a  student 
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leaves  the  class,  he/she  is  ready  to  use  VERT 
at  his/her  duty  station,  but  VERT  is  not 
available  everywhere.  The  shift  from  VERT 
will  give  the  student  tools  that  are  not  so 
computer  dependent  and  more  readily  used. 

To  handle  the  request  for  more  VERT  in¬ 
struction,  we  plan  to  offer  a  1-week  VERT 
seminar.  This  course  will  cover  more  VERT 
features  than'  are  presently  taught  in  any 
government  course.  Due  to  the  present  em¬ 
phasis  on  the  Total' Risk  Assessing  Cost  Esti¬ 
mate  (TRACE)  procedure,  this  course  shows  how 
VERT  can  be  used  to  do  a  TRACE  study.  This 
course  will  be  offered  on  a  trial  basis  about 
a  year  from  now  to  see  if  there  is  a  demand 
for  it. 

ALMC  is  also  looking  at  ways  to  improve 
the  input  of  data  into  the  VERT  program.  VERT 
requires  formatted  cards  or  formatted  card 
images  as  input  which  are  a  challenge  to  work 
with.  We  are  studying  the  possibility  of  us¬ 
ing  highly  portable  micro-computers  that  can 
be  used  to  elicit  data  from  students,  store  it 
formatted  for  VERT  input  and  transmit  it  to  a 
host  computer  for  processing.  When  this  be¬ 
comes  available,  it  would  help  ease  the  prob¬ 
lem  with  data  entry  which  discourages  all  but 
the  most  persistent  from  using  VERT. 

CONCLUSIONS 

Thirteen  years  have  gone  by  since  risk 
was  identified  at  the  highest  level  of  000  as 
a  cause  for  cost  growth,  schedule  slippages 
or  performance  degradation.  The  problems 
still  persist.  Will  training  solve  the  prob¬ 
lem?  I  really  cannot  answer  that  yet,  but 
whatever  the  solution  is,  training  will  have 
to  be  a  part  of  it  and  tie  Army  through  ALMC 
is  ready. 
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The  workshop  dealt  primarily  with  the  uncertainties  we  perceive  today.  This  talk  looked  to  the 
future  -  what  uncertainties  will  we  have  to  address  in  acquiring  future  syste  . 
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PROBLEMS  IN  IMPLEMENTING  RISK  TECHNIQUES 
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[SUMMARY  NOTES  FROM  DR.  HURTA' S  PRESENTATION] 


1.  There  is  confusion  over  the  meaning  of  terms'  (risk,  uncertainty,  etc.)  not  only  outside  th 
analytical  community  but  within  it.  And  although  there  may  be  some  merit  to  leaving  things  unde 
■fined,  the  harm  far  outweighs  the  good  (if  there  is  any).  Remember  Humpty  Dumpty:  "When  I  us 
a  word,  it  means  what  I  choose  it  to  mean,  neither  more  nor  less."  If  we  can't  agree  on  defini 
tions,  at  least  let's  define  our  terms  in  a  common  sense  way  whenever  we  speak,  write,  repor 
..  .on  our  subject. 

2.  As  a.  community  we  must  improve  our  communication  skills  -  to  be  able  to  relate  better  t 
each  other,  but  much  more  importantly  to  the  manager/s  decision  makers  we  prefer  to  help.  This  i 
crucial.  We  must  be  able  to  translate  our  expertise  into  terms  managers/decision  makers  can  easi¬ 
ly  understand.  Don't  expect  or  wait  for  them  to  learn  our  language  -  the  burden  is  on  us  to  lear 
theirs. 

3.  Back  off  ever  so  slightly  on  the  precision  we  may  desire  in  our  results.  Increase  the  effi¬ 
ciency  of  our  processes  which  maintaining  sufficient  effectiveness.  Don't  gold  plate  the  analy¬ 
sis.  It's  better  to  be  approximately  right  than  exactly  wrong. 

I  believe  that  consideration  of  items  2  &  3  would  tend  to  encourage  confidence  on  the  part  o 
managers/decision  makers  in  the  analyst  and  analytic  process. 

4.  We  must  become  more  manager/decision  maker-criented .  We  must  remember  that  our  work  usually 
is  not  on  end  in  itself  (our  end),  but  rather  a  means  to  an  end  (his). 

5.  We  should  work  to  bring  the  manager/decision  maker  more  into  the  process.  Most  of  the  pro¬ 
blems  we  work  on  aren't  ours,  they  are  theirs.  This  must  be  accomplished  through  a  more  humble 
interpretation  of  our  role  as  well  as  by  pursuading  the  manager/decision  maker  how  crucial  his 
role  is  to  the  analytical  process. 

6.  Tne  state  of  the  art  of  our  theories  has  far  outstripped  their  implementation.  Implementatior 
is  where  the  next  great  payoffs  are.  Although  great  progress  comes  from  invention,  I  believ 
that  far  more  progress  comes  from  thinking  of  applications  for  or  adapting  inventions. 

7.  Learn  some  new  skills.  Don't  tend  to  do  tomorrow  what  you  did  yesterday,  just  because  you  die 
it  yesterday.  No  model/tool  can  be  used  on  all  problems. 

8.  Find  out  what  tool  the  manager/decision  maker  likes,  or  is  comfortable  with.  This  may  cal' 
upon  you  to  u'e  a  technique  which,  from  the  analytical  perspective,  is  not  the  best  one.  But. 
remember,  there's  more  than  the  analytical  perspective. 


PAPERS  NOT  PRESENTED 


(SOME  HIGH  QUALITY  PAPERS  WERE  RECEIVED  THAT,  FOR  VARIOUS 
REASONS,  DID  NOT  FIT  INTO  THE  REGULAR  PROGRAM.  WE  ARE  PROUD 
TO  PRESENT  THEM  HERE.) 


AD  P002321 


DECISION  RISK  ANALYSIS  (DRA)  FOR  THE  REMOTELY  PILOTED  VEHICLE  (RPV) 

William  E.  Bodden 


'I 

j 

I 

i 

I 


U.S.  ARMY  AVIATION  RESEARCH  AND  DEVELOPMENT  COMMAND 
DEVELOPMENTAL  SYSTEMS  ANALYSIS  DIVISION 
4300  Goodfellow  Boulevard 
St.  Louis,  Missouri  63120 


ABSTRACT 

■s* 

The  topic  of  this  is  the  DRA  performed  to  sup¬ 
port  the  Army’s  RPV  Development  Program.  The 
basic  analytical  tool  utilized  was  the  Army’s 
Venture  Evaluation  and  Review  Technique  (VERT) 
computer  model.  The  paper  addresses  the 
general  methodology  utilized  in  performing  the 
DRA,  including  (1)  a  brief  discussion  of  VERT, 
(21  procedures  used  to  identify  risk  areas,  and 
collect  the  associated  data  necessary  to  assess 
their  impact,  and  (3)  problems  encountered  in 
adapting  certain  peculiar  situations  to  the 
VERT  methodology.  It  concludes  with  lessons 
learned,  f  _ 


The  Directorate  for  Plans  and  Analysis, 
AVRADCOM,  was  tasked  to  perform  a  Decision  Risk 
Analysis  to  determine  the  performance,  schedule, 
and  cost  risks  inherent  in  the  Remotely 
Piloted  Vehicle  Research  and  Development  (R&D) 
Program.  In  addition,  the  project  manager 
requested  that  the  analysis  results  be  utilized 
to  generate  a  year-by-year  TRACE  estimate. 

TRACE  stands  for  "Total  Risk  Assessing  Cost 
Estimate"  and  should  represent  "....  expected 
total  cost  over  a  specified  period  of  a 
materiel  development  program  ...  Including 
specific  provision  for  the  statistical 
estimation  of  probable  program  costs  otherwise 
indeterminate.  TRACE  should  be  that  estimate 
having  a  50/50  chance  of  producing  either  a 
cost  overrun  or  an  underrun." 

In  approaching  the  problem,  it- was  determined 
that  the  areas  of  performance  risk  were  too 
diversified  to  attempt  a  quantitative  assess¬ 
ment;  thus  a  qualitative  approach  was  used. 

Some  42  system  parameters,  ranging  from  the 
boresight  accuracy  of  the  FLIR  to  the  System 
Displacement  time  were  evaluated  by  system 
engineers.  The  vehicle  used  was  a  Risk  Assess¬ 
ment  Questionnaire.  For  each  parameter,  the 
engineer  was  asked  to  provide:  a  risk  level, 
a  brief  explanation  for  the  rating,  a  list  of 
problem  areas,  and  a  "fallback"  position 
(should  the  risk  prove  to  be  unacceptable)  . 
While  this  approach  has  limited  decision 
making  value,  it  is  useful  for:  providing  a 
good  "over-all  view",  identifying  problem 
areas,  and  putting  the  various  risk  areas  in 
proper  perspective.  * 

In  evaluating  schedule  and  cost  risks,  the 
Army's  VERT  model  was  used.  VERT  is  a  simu¬ 
lation  model  designed  for  network  evaluation. 


The  model  requires  construction  of  a  basic  flov 
network  consisting  of  nodes  (Milestones/normal 
activity  completion  points)  and  arcs  (activi¬ 
ties)  initiating  and  terminating  at  nodes. 

Each  arc  can  be  assigned  three  parameters  -  a 
time,  a  cost  and  a  performance  parameter. 
Generally,  the  time  and  cost  parameters  repre¬ 
sent  the  resources  consumed  in  performing  the 
activity,  while  the  performance  parameter 
represents  the  "capability"  gained  for  that 
expenditure. 

As  activities  are  completed  (or  partially  com¬ 
pleted)  their  resource  expenditures  and  per¬ 
formance  gains  are  accumulated.  Thus,  for  each 
pass  through  the  network  (each  iteration  in  the 
simulation),  total  cost,  time,  and  performance 
are  determined. 

Each  node  is  assigned  an  input  and  output  logic 
-  the  input  logic  setting  the  conditions  on 
the  incoming  arcs  that  must  be  met  before  the 
output  logic  is  executed,  and  the  output  logic 
determining  which  outgoing  activities  are 
triggered.  Examples  of  logic  are: 

AND:  All  incoming  activities  must  be  complete^ 
before  the  output  logic  is  initiated. 

PAND:  All  incoming  activities  must  be  either 
completed  or  terminated  (with  at  least  one 
completed)  before  output  logic  is  Initiated. 

OR:  One  completed  incoming  activity  initiates 
output  logic  without  waiting  for  a  determina¬ 
tion  on  the  remaining  incoming  activities. 

Examples  of  output  logic  are: 

ALL:  All  outgoing  activities  are  triggered. 

MONTE  CARLO:  Each  outgoing  activity  is  assign 
ed  a  probability  of  occurence  (total  l)  and  th 
activity  triggered  is  chosen  on  a  probabilisti 
basis  by  generation  of  a  random  number. 

FILTER:  Each  outgoing  activity  is  assigned  a 
range  of  values  for  time,  cost  and/or  perform¬ 
ance.  If  the  respective  parameter  value(s) 
accumulated  at  this  point  in  the  network  lie 
within  these  ranges,  the  activity  is  triggere< 

Nodes  may  also  be  assigned  what  is  called  a 
unit  logic,  rather  than  separate  input  and 
output  logic.  These  nodes  have  N  input  arcs 
wnich  mate  with  one  of  N  output  arcs  to  enabl> 
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direct  transmission  of  the  network  flow  from  a 
given  input  arc  to  a  given  output  arc. 

Nodes,  in  themselves,  are  not  assigned  specific 
time/cost/performance  values  (since  they  are 
not  activities);  however,  the  model  does 
accumulate,  at  each  node,  the  cotal  time 
expended,  cost  incurred  and  performance  gener¬ 
ated  to  process  all  the  arcs  encountered  along 
the  path  the  network  flow  followed  in  order  to 
complete  the  processing  of  the  node. 

Time  is  the  basic  parameter  controlling  net¬ 
work  flow,  and  it  determines,  not  only  the 
values  accumulated  at  each  node  but  other 
pertinent  information  such  as  critical  path. 

The  data  (time,  cost,  performance)  for  each 
activity  can  be  assigned  in  four  ways: 

(1)  Direct  Input 

(2)  A  mathematical  relationship  involving 
either  data  from  previously  processed  arcs  or 
nodes  or  the  remaining  data  for  the  activity 
itself. 

(3)  As  a  random  variable  or 

(4)  A  combination  of  (1)  thru  (3)  . 

This  maximizes  the  realism  that  can  be  built 
into  the  network.  VERT  provides  37  trans¬ 
formations  to  aid  in  structuring  mathematical 
relationships  and  14  statistical  distributions 
to  facilitate  the  modeling  of  random  variables. 

Other  features  which  make  VERT  a  very  flexible 
analytic  tool  are : 

(a)  Use  of  a  random  number  seed  which  allows 
duplication  of  results 

(b)  Histogram  input  capability 

(c)  Printout  of  intermediate  results 

(d)  A  pruning  option  which  permits  accumula¬ 
tion  of  partial  costs 

(e)  Accumulation  of  data  at  pre-selected  time 
intervals 

(f)  Inflation/Discount 

(g)  Detailed,  explicit  error  messages 

In  establishing  the  basic  network  for 
evaluating  the  RPV  schedule  and  cost  risk, 
technical  personnel  provided  a  flow  diagram 
identifying  the  major  technical  milestones  for 
each  subsystem,  as  well  as  a  low,  most  likely, 
and  high  estimate  of  time  required  to  complete 
each  activity.  For  lack  of  better  information, 
each  activity's  time  was  treated  as  a  random 
variable  using  triangular  distribution.  '  In 
addition  to  the  technical  aspects  of  the  R&D 
effort,  the  support  (ILS)  impact  on  schedule 


was  also  considered.  Since  the  IES  effort  did 
not  necessarily  coincide  with  the  technical 
flow,  it  was  decided  to  evaluate  where,  in  the 
basic  network,  the  IES  effort  could  affect  the 
schedule;  this  impact  was  then  addressed  as  a 
low,  most  likely,  and  high  schedule  slippage 
anticipated  at  that  point  due  to  IES  problems. 
Again  a  triangularly  distributed  random 
variable  was  assumed.  Using  1000  iterations,  a 
cumulative  distribution  of  probable  months  to 
complete  the  R&D  effort  was  derived.  This 
distribution  was  used  to  analyze  the  probabil¬ 
ity  that  the  program  would  be  completed  within 
the  time  frame  expected  by  the  Project  Manager. 
We  also  provided  the  mean  (expected  value)  , 
median  (50/50  chance) ,  and  "90%  confidence" 
times-to-coraplete  to  the  PM. 

For  assessing  cost  risk,  the  PM's  Baseline 
Cost  Estimate  (BCE)  was  used  to  represent  the 
most  likely  cost  associated  with  his  most 
likely  schedule.  Cost  analysis  personnel  were 
then  asked  to  quantify  cost  risks  in  three 
different  areas: 

(1)  Cost  uncertainty  -  those  costs  associated 
strictly  with  potential  estimating  error  and 
involving  no  change  in  scope  of  work  nor 
schedule  impact. 

(2)  Fixed  Cost  Growth  -  Costs  associated  with  a 
change  in  the  scope  of  work  which  is  not  time 
sensitive  (e.g.,  adding  an  additional  training 
course  to  be  conducted  simultaneously  with 
those  scheduled) ,  and 

(3)  Variable  Cost  Growth  -  Additional  costs 
resulting  from  schedule  slippage.  In  addres¬ 
sing  cost  uncertainty,  a  high  and  low  factor 
(to  be  applied  to  the  BCE  estimate)  were 
obtained  for  each  major  subsystem.  This  was 
input  to  the  model  and,  using  a  triangular 
distribution,  resulted  in  a  single  "distribu¬ 
tion  factor"  for  each  subsystem  and  each 
iteration.  This  factor  was  then  applied  to  the 
BCE  estimate  (as  well  as  the  fixed  and  variable 
cost  growth  estimate)  to  generate  the  cost 
associated  with  that  particular  iteration. 

Fixed  cost  growth  values  were  directly  input 
to  the  model  (at  the  appropriate  point) , 
utilizing  arcs  with  zero  time.  Variable  cost 
growth  was  addressed  on  an  arc  by  arc  basis  by 
applying  a  "cost  per  month"  (for  every  month 
beyond  the  most  likely)  using  a  mathematical 
relationship. 

Again  1000  iterations  were  run,  and  cumulative 
cost  distributions  were  derived  for  the  total 
program,  as  well  as  (using  the  model's  parti¬ 
tioning  and  pruning  options)  each  fiscal  year. 

These  results  were  then  used  to  address  cost 
risk  and  develop  the  year-by-year  TRACE 
estimate. 


In  performing  the  DRA,  several  situations  arose 
which  required  some  manipulation  of  the  model 
to  achieve  realism. 

In  trying  to  run  both  schedule  and  cost  risk 
simultaneously,  it  was  necessary  to  introduce  a 
"BCE  Cost  Path"  with  times  incremented  at  12 
month  intervals  (fiscal  years).  To  prevent 
this  path  from  dominating  the  schedule  errone¬ 
ously  (in  those  instances  where  the  program  may 
come  in  under  schedule),  the  total  path  time 
was  subtracted  at  the  very  end.  But  the  model, 
being  time  driven,  will  subtract  cost  whenever 
it  backtraeks  in  time.  This  then  would  mis¬ 
represent  the  cost  assessment.  This  dilemma 
could  have  been  avoided  if  the  BCE  estimate 
could  have  been  broken  down  by  activity; 
however,  such  detail  was  hot  available  so  that 
cost  and  schedule  had  to  be  run  separately. 

In  superimposing  ILS  schedule  impact  on  the 
basic  technical  network,  it  was  necessary  in 
some  cases  to  give  the  ILS  impact  "credit"  for 
schedule  slippage  already  assessed  for  techni¬ 
cal  problems.  This  again  necessitated  using 
negative  time.  To  avoid  a  cost  problem,  a 
dummy  arc  was  inserted  in  the  network  parallel 
to  the  ILS  portion.  This  insured  that  the  time 
"coming  out"  of  this  section  was  never  less 
than  the  t  "going-in",  yet  the  use  of  nega¬ 
tive  time  still  properly  diminished  the  ILS 
Impact  when  appropriate. 

A  final,  though  minor,  problem  concerned  cost 
distribution  factors.  Since  all  cost  risk  was 
evaluated  using  basic  BCE  data  and  since  the 
BCE  methodology  did  nor.  logically  get  any 
better  or  worse  by  fiscal  year,  it  was  felt 
that  a  single  uncertainty  distribution  should 
be  used  (for  each  subsystem)  rather  than  an 
independent  one  for  each  activity  and/or  fiscal 
year.  This  was  accomplished  by  means  of  a 
(previously  unused)  performance  arc  early  in 
the  network. 

In  performing  the  RPV  DRA,  I  came  to  several 
conclusions : 

(1)  The  VERT  is  an  excellent,  highly  flexible, 
realistic  tool  for  risk  assessment. 

(2)  The  DRA  is  a  valuable  management  tool. 

This  particular  effort  resulted  in  the  Identi¬ 
fication  of  potential  schedule  problems  and 
budgetary  inconsistencies . 

(3)  To  continue  to  provide  the  PM  with  thi s 
type  of  assistance,  it  should  be  updated  r erai- 
annually.  I  am  pleased  to  say  that  this  i.;  the 
current  intent  of  the  RPV  Project  Manager. 

I  also  formulated  a  philosophy  for  "selling" 
the  DRA  concept  to  reluctant  managers: 

(a)  Realize  that,  at  least  initially,  the 
relationship^will  be  adversarial;  this  is 
normal  -  the  analyst  is  looking  for  problem 


areas,  the  program  manager  doesn't  like  problem 
areas . 

(b)  Keep  everything  out  in  the  open  -  explain 
exactly  what  you  are  trying  to  do,  what  data 
you  need,  and  exactly  how  it  will  be  used. 

(c)  If  possible,  deal  with  a  supervisor  common 
to  those  providing  data.  This  makes  conflict 
resolution  much  easier. 

(d)  Make  sure  the  data  collected  realistically 
reflects  the  situation  being  modeled;  then  let 
the  data  determine  the  results.  Frequently 
the  temptation  is  to  adjust,  and  rationalize, 
the  data  in  such  a  way  that  preconceived 
results  are  obtained. 

(e)  Most  important  -  have  a  firm  conviction  of 
the  value  that  such  an  analysis  can  be  to  the 
project  manager. 
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INTRODUCTION 


The  purpose  of  this  article  is  to 
share  the  preliminary  results  of  a 
schedule  analysis  performed  on  a  major 
Air  Force  systems  acquisition  program. 
The  program  is  in  Full  Scale  Development 
(FSD) .  Portions  of  the  FSD  program  are 
concurrent  with  the  production  program. 
The  time  span  for  both  developing  and 
manufacturing  qualification  test 
hardware  is  shorter  than  normal.  The 
schedule  analysis  was  performed  with  the 
assistance  of  a  computer  program,  called 
PREDICT  2000.  PREDICT  2000  enabled  both 
engineers  and  managers  to  more 
accurately  quantify  their  experiences  so 
that  a  working  schedule  could  be 
established  prior  to  the  development  of 
the  contractor's  formal  scheduling 
system. 


The  results  of  the  schedule 
analysis  were  used  by  the  Air  Force 
contractor  to  modify  initial  schedules 
and  change  the  make/buy  structure  to 
meet  overall  program  milestones.  While 
the  final  results  of  this  approach  will 
not  be  known  until  January,  1984,  the 
preliminary  results  look  quite  good  and 
It  appears  that  the  approach  used  in 
this  paper  may  be  a  key  factor  in  the 
contractor  achieving  contractual 
milestones  which  would  have  otherwise 
been  missed,  c- 


C0NCURRENCY,  RISK  AND  UNCERTAINTY 


The  beginning  of  a  major  program 
tends  to  be  the  point  of  maximum  risk. 
The  engineering  design  is  normally 
incomplete  and  the  management  systems 
which  will  be  used  to  control  the 
program  are  in  the  process  of  being 
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developed.  These  factors  tend  to 
increase  the  risk  and  uncertainty 
associated  with  a  systems  acquisition. 
However,  it  is  also  during  the  initial 
stages  of  a  program  that  the  most 
flexibility  exists  to  make  changes  in 
make/buy  strategies  and  internal 
schedules  so  that  overall  program 
milestones  can  be  met. 


The  risk  inherent  in  the  beginning 
of  a  program  can  be  Increased  by  program 
concurrency.  Many  D0D  programs  face  a 
substantial  amount  of  concurrency 
because  it  is  necessary  to  shorten  the 
time  span  from  the  conceptual  idea  to 
the  fielded  system.  Concurrency  makes 
it  necessary  to  release  engineering 
designs  before  they  have  been  completely 
proven.  The  press  to  design  a  product 
meeting  specifications  tends  to  reduce 
the  interaction  among  the  manufacturing 
and  the  design  engineers  which  assures  a 
producible  product.  The  lack  of  mature 
engineering  designs  also  reduces  the 
ability  of  manufacturing  engineers  and 
managers  to  plan,  control,  and  assess 
the  manufacturing  processes,  costs  and 
schedules.  Risk  and  uncertainty  is 
quite  frequently  compounded  because  it 
takes  several  months  to  input  data  into 
the  management  systems  so  that  they  can 
work  effectively.  Thus,  scheduling 
early  in  a  program  is  usually  based  upon 
parametric  approaches  or  experience. 


An  Air  Force  parametric  analysis  of 
historical  data  for  similar  programs 
showed  that  there  was  a  low  likelihood 
that  the  contractor  would  successfully 
achieve  the  schedule  for  completing  full 
scale  development (FSD)  on  this  program. 
The  FSD  schedule  is  driven  by  the  stated 
initial  operating  capability( IOC)  date. 
The  IOC  date  must  be  achieved  for  the 
acquisition  program  to  be  considered  a 
success.  Thus,  it  was  necessary  for  the 


program  office  to  emphasize  the  need  for 
good  schedule  analysis  early  in  the 
program.  In  mid-Novemher ,  five  months 
after  the  completion  of  source 
selection,  the  time  for  completing  the 
engineering  design  of  the  62  most 
critical  parts  in  the  acquisition 
program  was  analyzed  using  PREDICT  2000. 
The  input  for  PREDICT  2000  was  provided 
by  the  design  engineers  responsible  for 
completing  the  design.  Prior  to  the 
analysis  using  PREDICT  2000,  the  company 
had  established  a  schedule  for 
completing  the  design  of  62  critical 
parts;  this  schedule  had  estimated  that 
the  design  would  be  complete  by 
mid-January.  The  analysis  using  PREDICT 
2000  estimated  the  design  completion  the 
end  of  the  last  week  in  February.  The 
last  week  in  February  was  identified  by 
the  Air  Force  in  November  as  the  time 
frame  for  conducting  a  schedule  analysis 
of  the  remaining  manufacturing  tasks; 
the  results  of  that  analysis  will  be 
discussed  in  the  last  section  of  this 
paper.  With  the  exception  of  four  items 
which  had  to  be  redesigned,  the  design 
of  the  critical  items  wa3  completed  by 
the  last  week  in  February.  It  appears 
that  the  engineers'  ability  to  estimate 
design  times  was  substantially  improved 
using  the  computer  aid. 


PREDICT  2000-  AN  AID  IN  ESTIMATING 

PREDICT  2000  solves  a  major 
operations  research  problem  which 
concerns  how  one  generates  an 
appropriately  shaped  probability  density 
function  when  only  sparse  data  is 
available.  PREDICT  2000  can  generate 
cumulative  probability  distributions 
based  upon  the  thirty-seven  different 
generic  shapes  shown  in  Figure  1  from 
experiential  data.  Although  each  of  the 
twenty-five  group  II  shapes  is  shown 
unskewed  (that  is,  50*  of  the  function 
is  on  each  side  of  the  mode),  each  shape 
can  assume  any  level  of  skewness. 

The  PREDICT  2000  computer  software 
Interacts  with  the  user  through  a 
mini-computer.  It  uses  one  measure  of 
central  tendency  (the  mode)  and  two 
measures  of  dispersion(a  100*  percentile 
range  and  another  range  specified  by  the 
user)  to  provide  its  output.  The  mode 
was  chosen  as  the  PREDICT  2000  measure 
of  central  tendency  both  because  people 


can  estimate  the  mode  more  accurately 
than  the  mean  and  because  the  mode  has 
analytical  importance  in  determining  the 
probability  density  function's  shape. 

The  ranges  are  also  easy  for  people  to 
estimate  and  provide  a  basis  for 
determining  the  shape  of  the  probability 
function.  In  summary,  PREDICT  2000 
requires  eight  pieces  of  data  to  provide 
an  output: 

(1)  The  variable's  name  which  is 
used  to  identify  the  output. 

(2)  The  variable's  dimension  which 
is  used  to  label  the  output. 

(3)  The  user  defined  percentile 
range.  This  percentile  range  is 
symbolized  by  R  and  lies  between  the 
points  R^^  and  R2-  These  points  are 

identified  by  the  user.  The  value  of 
R  is  used  to  define  the  area  below  R-^ 

and  above  R2>  N  has  the  value  of 

(R+( (l-R)/2)xl00)*. 

(4)  The  lowest  possible  value. 

(5)  The  value  associated  with  R. 

I* 

This  is  the  value  above  which  N*  of  the 
distribution  is  found. 

(6)  The  most  likey  value(mode). 

(7)  The  value  associated  with  R2. 

This  is  the  value  below  which  N*  of  the 
distribution  is  found. 

(8)  The  highest  possible  value. 


Given  this  information,  PREDICT 
2000  generates  the  complete  cumulative 
probability  function  and  calculates  the 
mean  value  for  the  function.  Normally, 
when  addressing  the  problem  of 
scheduling  a  production  program,  it  is 
the  mean  value  which  one  desires  to  use. 

The  determination  that  the  design 
would  be  complete  the  last  week  in 
February  was  based  upon  the  mean  time  tc 
complete  design.  However,  the  same 
analysis  also  considered  the  likelyhood 
of  values  above  the  mean.  The  four 
items  whose  design  had  not  been 
completed  fell  into  the  risk  range 
identified  in  the  analysis. 
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ANALYZING  THE  MANUFACTURING  SCHEDULES 

The  development  of  an  estimate  of 
when  the  engineering  would  be  complete 
was  not  a  matter  of  idle  curiosity.  The 
manufacturing  tasks  can  not  be 
successfully  begun  until  the  majority  of 
the  engineering  is  complete.  This 
causes  what  is  often  referred  to  as  the 
bow  wave  effect.  The  majority  of  the 
engineering  tasks  are  completed  at  about 
the  same  time.  Upon  their  completion, 
the  manufacturing  planners,  schedulers, 
tool  designers,  and  procurement 
specialists  can  begin  their  efforts  to 
turn  the  design  into  hardware.  The 
completion  of  the  engineering  tasks  at 
nearly  the  same  time  causes  the  next 
tier  of  tasks  for  each  new  specialty  to 
be  grouped  together.  If  the  overall 
schedule  is  tight,  then  this  bow  wave 
will  continue  onto  the  production  floor. 
Thus,  a  slip  in  the  engineering  design 
schedule  can  cause  problems  throughout 
the  entire  program.  Moreover,  the 
manufacturing  tasks  which  result  in  the 
production  of  the  hardware  can  not  be 
begun  until  the  design  for  each  piece  of 
hardware  is  complete. 

The  sixty-two  parts  which  were 
identified  as  critical  were  parts  which 
are  typically  found  on  the  critical  path 
of  PERT/CPM  networks  of  this  type  of 
product.  Experience  had  shown  that 
those  parts  were  the  schedule  drivers. 
Thus,  they  were  chosen  as  the  parts  to 
track  prior  to  the  development  of  a 
formal  tracking  system. 

It  should  also  be  noted  that  this 
contractor  has  an  excellent  formal 
scheduling  system.  However,  this  system 
was  not  fully  established  until  June 
1983.  The  time  frame  from  March  through 
May  1983  was  crucial  for  making  schedule 
adjustments.  The  schedule  analysis 
using  PREDICT  2000  showed  that  all  of 
the  available  slack  time  would  have  been 


gone  and  the  schedule  would  be 
unattainable  for  all  practical  purposes 
if  the  analysis  had  been  deferred  until 
June  1983. 

PREDICT  2000  was  used  to  estimate 
the  mean  times  for  procurement  of 
material,  manufacturing  operations  and 
tooling  sheet  development  and  tool 
manufacturing,  and  part  fabrication. 
Parts  had  to  be  fabricated  by  the  end  of 
September  to  meet  a  November  internal 
schedule.  The  contractual  date  for  the 
first  unit  to  begin  qualification  test 
was  the  end  of  December.  The  time 
allowable  for  concurrency  between 
material  procurement  and  fabrication  and 
manufacturing  operations  sheet 
development  and  tooling  manufacturing 
and  fabrication  was  determined  and 
included  in  the  analysis.  The  final 
result  of  the  analysis  was  depicted  on  a 
traditional  Gantt  chart  showing 
procurement  of  materials,  manufacturing 
planning,  and  fabrication  of  the  parts. 
One  month  was  allocated  for  assembly. 

The  desired  schedule  was  planned  to 
include  contingency  time  of  between  one 
and  two  months.  The  analysis  shown  in 
Figure  2  provided  the  contractor  with 
insights  into  the  overall  schedule  at 
the  end  of  February  1983  which  would 
otherwise  not  have  been  available  until 
June  1983.  The  contractor  took 
management  action  based  upon  the  PREDICT 
2000  schedule  analysis  which  resulted  in 
an  adjusted  schedule  meeting  the 
contractual  requirements.  The  PREDICT 
2000  schedule  analysis  took 
approximately  three  people  three  mandays 
to  accomplish. 

Currently,  the  program  appears  to 
be  on  schedule.  The  final  proof  of  the 
merit  of  this  approach  will  be  if  the 
contractor  can  actually  build 
qualification  test  article  on  schedule. 
The  preliminary  results  do  suggest  that 
PREDICT  2000  can  improve  the  quality  of 
schedule  estimates. 
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Meridian  Corporation  has  developed  an  approach 
to  risk  assessment  'and  contingency  reserve 
allocation  which  draws  upon  numerous  statisti¬ 
cal  and  empirical  techniques  to  evaluate 
contractor  performance.  The  approach  is 
motivated  by  the  need  to  anticipate,  at  the 
highest  levels  of  program  management,  poten¬ 
tial  cost  overruns  so  that  corrective  or 
pre-emptive  actions  may  be  initiated  early  in 
the  development  cycle.  This  paper  discusses 
the  nature  of  the  analytical  toola  used  to 
assess  the  risk  of  cost  growth  and  describes 
the  application  of  these  tools  to  an  indivi¬ 
dual  DARPA  program. 

The  approach  presented  in  this  paper  is 
Intended  to  supplement  rather  than  to  sup¬ 
plant  traditional  methods  of  contract  cost 
analysis.  It  is  oriented  toward  the  needs  of 
senior  level  decision-makers  who  must  evaluate 
in  the  aggregate  the  requirements  for  contin¬ 
gency  reserves  and  who  have  ultimate  respon¬ 
sibility  for  the  successful  completion  of 
the  program  within  established  cost,  schedule, 
and  technical  constraints.  It  is  designed  to 
be  used  in  conjunction  with  the  more  detailed 
management  analyses  of  specific  projects  to 
illuminate  trends  in  cost  growth  and  to 
provide  a  comparison  to  heuristic  models  and 
empirically-derived  cost  distributions. 

The  approach  consists  of  a  series  of  risk'^'v^ 
assessment  indicators  which  collectively 
address  the  potential  for  short-term,  mid¬ 
term,  and  long-term  cost  growth.  The  outputs 
range  from  short-term  projections  of  an 
"envelope"  of  future  costs  by  WBS  element  to 
long-term  estimates  of  total  contract  cost 
and  probabilities  of  attaining  budget  targets. 
Finally,  results  obtained  from  applying  this 
approach  to  an  actual  DARPA  program  are 
presented.  Scenarios  for  how  the  outputs  can 
be  assimilated  into  the  program  review  and 
budget  allocation  process  are  described,  and 
Issues  to  be  considered  in  the  Implementation 
of  the  approach  are  identified. 

OVERVIEW  OF  PROGRAM  RISK  MANAGEMENT  TECHNIQUES 

The  general  problem  of  assessing  risk  and  un¬ 
certainty  in  major  programs  has  received 
considerable  attention  in  the  risk  analysis 
literature.  Numerous  attempts  have  been  made 
to  analyze  the  reasons  for  cost  overruns, 
although  contributions  to  the  state-of-the-art 
in  risk  analysis  have  originated  from  a  wide 


Meridian's  review  of  the  state-of-the-art  in 
risk  analysis  undertaken  at  the  outset  of  the 
project  has  yielded  some  Interesting  inaights 
into  the  thrust  of  previous  efforts.  Previous 
publications  concerning  cost  growth  risk  may 
be  broadly  characterized  as  falling  into  one 
of  three  categories: 

0  Analyses  of  structural  deficiencies  which 
indirectly  cause  cost  overruns. 

0  Analyses  of  causal  factors  which  were 
directly  responsible  for  incurring  cost 
overruns . 

0  Descriptions  of  models  which  attempted  to 
assess  the  likelihood  and  impact  of  cost 
overruns. 

None  of  the  previous  analyses  adequately 
addresses  the  particular  circumstances  and 
requirements  of  the  DARPA  Program  Management 
Office  (PMO),  although  valuable  lessons  can  be 
learned  through  the  study  of  such  analyses. 
Analyses  undertaken  in  each  of  the  three 
groups  of  analysis  are  summarized  in 
Meridian's  final  report  to  DARPA.  (see 
Reference) 

Relationship  to  PMO  Requirements 


From  the  outset,  a  primary  concern  of  the  PMO 
was  the  need  for  a  mechanism  to  gauge  the 
likelihood  of  a  project  requiring  additional 
funds  due  to  unforeseen  circumstances  and  to 
assess  the  magnitude  of  such  additional  funds. 
Such  requirements  had  arisen  historically  and 
had  a  significant  disruptive  impact  on  DARPA' s 
ability  to  pursue  its  mission  successfully. 
However,  the  role  of  FMO  analysts  is  broad 
program  oversight  and  budgetary  control,  and 
the  analyses  undertaken  at  the  PMO  level  are 
generally  oriented  toward  financial  rather 
than  technical  management.  Consequently,  in¬ 
formation  of  a  technical  nature  is  of  limited 
utility  since  engineering  trade-off  analyses 
are  the  predominant  responsibility  of  the 
Technical  Offices,  which  also  have  responsi¬ 
bility  for  technical  risk  assessment.  As  a 
result,  the  inputs  which  may  be  used  in  PMO 
risk  assessment  must  be  limited  to  information 
which  is  readily  available  st  the  PMO  and  the 
risk  assessment  mechanism  applicable  to  a 
diverse  set  of  projects  and  technologies. 

Also,  it  was  recognized  that  any  mechanism 
proposed  to  assist  in  identifying  potential 
program  risk  areas  must  be  sensitive  to  the 
workload  already  imposed  on  PMO  analysts. 
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Therefore,  the  PMO  risk  assessment  procedure 
must  be  easily  administered  by  existing  staff 
members  without  requiring  excessive  data  col¬ 
lection  and  reduction  efforts. 

In  developing  procedures  for  use  by  DARPA  to 
address  these  general  agency  requirements,  two 
sets  of  characteristics  were  identified  — 
conceptual  and  organizational.  These  charac¬ 
teristics  represent  the  basis  for  the  structure 
of  the  risk  assessment  tools  described  in  later 
sections. 

Conceptual  characteristics  of  the  approach 
to  program  risk  assessment  include  the 
following: 

o  The  approach  must  be  sufficiently  general 
to  be  applicable  to  a  wide  range  of  hetero¬ 
geneous  projects. 

o  The  approach  should  utilize  data  available 
at  the  FMO  so  as  not  to  impose  additional 
reporting  requirements  on  contractors  or 
DARPA  agents . 

c  The  approach  should  be  relatively  easy  to 
implement  and  to  operate  on  a  regular 
basis. 

o  The  algorithms  should  ideally  be  transparent 
to  the  users,  preferably  being  contained  in 
a  "user-friendly"  interactive  minicomputer 
program. 

o  The  approach  should  permit  the  PMO  analyst 
sufficient  flexibility  to  interpret  results 
and  to  draw  independent  conclusions  con¬ 
cerning  future  risk  mitigation  alternatives. 

Organizational  characteristics  of  the  approach 
Include  the  following: 

o  The  approach  should  be  executable  by 
existing  PMO  staff  and  should  not  require 
additional  personnel  to  operate, 
o  The  approach  should  be  sufficiently  expedi¬ 
tious  to  permit  timely  interface  with  other 
budget  cycle  activities, 
o  The  approach  should  complement  rather  than 
supplant  the  project  management  activities 
of  the  Technical  Offices. 

As  mentioned  previously,  analyses  conducted  by 
Meridian  indicate  that  none  of  the  techniques 
Identified  as  currently  available  adequately 
address  the  special  needs  and  environment  of 
the  PMO.  An  examination  of  models  for  risk 
analysis  showed  that  most  were  oriented 
toward  program-specific  cost  growth.  Cost 
growth  models  of  this  genre  are  appropriate 
and  useful  for  the  management  of  programs  at 
the  level  of  the  Technical  Offices,  since 
technical  risk  and  cost  growth  risk  are 
generally  analyzed  by  component  in  these 
models  rather  than  in  the  aggregate.  Unfor¬ 
tunately,  the  detail  which  makes  these 
models  useful  at  the.  component  level  tends  to 
make  them  ineffectual  for  the  Program  Manage¬ 
ment  Office  due  to  their  specificity,  their 


cost,  and  the  amount  of  data  required  to  run 
them. 

For  all  these  reasons,  the  approach  focused 
on  the  identification  of  risk  assessment 
indicators  which  could  be  used  by  PMO  analyst 
to  gain  an  understanding  of  potentially  risky 
areas  at  a  higher  level  of  aggregation  than 
afforded  detailed  causal  models.  The  risk 
assessment  indicators,  while  not  designed  to 
provide  definitive  answers  concerning  even¬ 
tual  cost,  are  oriented  toward  revealing 
trends  in  costs  incurred  tc  date.  In  thi3 
manner,  areas  of  likely  cost  growth  can  be 
flagged  early  enough  to  permit  a  detailed 
examination  of  the  reasons  for  cost  growth  ar 
to  permit  intervention  by  PMO  analysts  and 
Technical  Offices  to  forestall  cost  overruns 
through  active  or  passive  risk  reduction 
measures. 

Several  analytical  techniques  were  investi¬ 
gated  by  Meridian  in  connection  with  the 
development  of  risk  assessment  tools. 

Meridian  has  characterized  these  tools  accord 
ing  to  the  time  horizon  that  costs  are  pro¬ 
jected  through,  i.e.  short  term,  mid  term,  an 
long  term.  Short  term  refers  to  a  period  les 
than  six  months  in  the  future.  Mid  term  pro¬ 
jections  are  for  a  period  1/2  to  1-1/2  years 
in  the  future,  and  long  term  refers  to  period 
greater  than  1-1/2  years.  A  description  of 
these  tools  and  how  they  may  be  applied  to  th 
analysis  of  DARPA  programs  follows. 

Through  the  analysis  of  techniques  with  poten¬ 
tial  applicability  to  DARPA  program  risk 
assessment,  it  became  apparent  that  no  single 
risk  assessment  tool  could  satisfy  all  of  th° 
requirements  of  the  PMO.  Some  tools  appeared 
to  be  valid  for  predicting  short-term  fluctua 
tions  in  project  costs,  but  lacked  accuracy 
for  long-range  forecasts.  Others  provided 
good  indications  of  likely  long  term  cost  per 
formance,  but  did  not  provide  acceptable  shor 
term  estimates.  The  methodology  presented  in 
this  paper  is  comprised  of  several  techni¬ 
ques,  each  oriented  toward  a  different  facet 
of  program  risk  assessment.  The  differing 
emphases  of  these  techniques  is  illustrated  i 
Figure  1. 

As  will  be  discussed  below,  these  techniques, 
while  individually  not  capable  of  addressing 
all  risk  assessment  concerns,  collectively 
provide  a  firm  indication  of  the  relative 
status  and  financial  risks  of  the  programs 
subjected  to  the  analysis.  The  theory  behim 
each  of  the  component  risk  assessment  indica¬ 
tors  will  be  described  individually  with 
reference  to  results  obtained  from  applying 
these  indicators  to  an  ongoing  DARPA  program. 

RISK  ASSESSMENT  INDICATORS 
As  indicated  in  Figure  1,  the  risk  assessment 
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Figure  1 

RISK  ASSESSMENT  INDICATORS 
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month  and  a  previous  time  period  (e.g.  last 
month,  three  months  ago,  project  initiation). 
The  contract  cost  status  indicators  are 
summarized  in  Figure  2  with  similar  indica¬ 
tors  developed  for  schedule  status. 

Utilizing  these  indicators,  the  CPA  model 
then  computes  estimated  costs  at  completion 
and  determines  the  variances  (absolute  and 
percentage)  between  those  estimates  and  the 
total  budget  at  completiou  plus  management 

Figure  2 
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indicators  can  be  distinguished  by  the  time 
horizon  of  their  applicability.  The  indica¬ 
tors  would  be  provided  to  the  program  analyst 
who  would  assess  the  requirements  for  short¬ 
term,  mid-term  and  long-term  risk  management 
activities.  In  order  to  facilitate  an  under¬ 
standing  of  how  the  risk  assessment  indicators 
will  be  utilized  collectively,  the  following 
sections  will  first  describe  the  rationale 
behind  each  of  the  indicators  separately. 

The  methodology  for  analyzing  the  collective 
•',!  output  of  the  indicators  is  then  presented. 

t  Cost  Performance  Analysis  (CPA)  Model 

The  Cost  Performance  Analysis  (CPA)  model  was 
developed  by  Captain  Douglas  Tyler  of  the  Air 
,  Force  Weapons  Laboratory.  It  is  currently 
operational  at  DARPA  and  is  significant  for 
several  reasons.  First,  the  model  uses  data 
from  the  contractor's  Cost  Performance  Report 
(CPR)  as  inputs  as  the  other  tools  Meridian 
developed  were  constrained  to  do.  Second, 
the  risk  assessment  tools  developed  by 
Meridian  were  Intended  to  complement  the 
existing  cost  analysis  procedures  already  in 
place  at  DARPA.  Therefore,  the  tools  that 
Meridian  developed  were  oriented  toward  pro¬ 
viding  information  not  adequately  addressed 
by  the  CPA  model.  As  a  result,  a  brief 
review  of  the  CPA  model  and  its  areas  of 
applicability  and  validity  is  necessary  as 
'  part  of  an  Integrated  analysis  of  DARPA' s 
program  risk  analysis  capabilities. 

The  CPA  model  is  an  automated  tool  resident 
on  DARPA' s  HP  3000  that  automatically 
generates  data  and  ratios  of  interest  to  the 
program  analyst.  In  addition,  the  model 
estimates  cost  st  completion  utilizing  trend 
extrapolation  of  a  variety  of  these  ratios. 

The  CPA  model  computes  a  number  of  contract 
status  indicators,  based  upon  inputs  from  the 
contractor's  monthly  reports,  primarily  in 
the  form  of  ratios  between  these  data  ele¬ 
ments  and  differences  between  the  current 
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reserves.  The  model  then  provides  an  assess¬ 
ment  of  the  condition  of  the  contract  (e.g., 
condition  red,  yellow,  green,  or  blue). 

The  estimates  at  completion  (EAC)  are  computed 
seven  ways,  each  of  which  gives  a  slightly 
different  indication.  The  seven  EAC  values 
are  based  on: 

o  contractor's  LRE  (Latest  Revised  Estimate) 
o  current  month  CPI  (Cost  Performance  Index) 
o  3  month  CPI 
o  cumulative  CPI 

o  ACWP  (Actual  Cost  of  Work  Performed)  regres¬ 
sion 

o  weighted  SPI/CPI  (Schedule  Performance 
Index/Cost  Performance  Index) 
o  trend  weighted  SPI/CPI 

The  sources  and  uses  of  the  seven  EAC  values 
are  summarized  in  Figure  3. 

The  CPA  model  has  been  used  by  DARPA  for  pro¬ 
gram  cost  analysis  for  some  time  and  is  useful 
for  certain  fundamental  program  management 
responsibilities,  including  both  accounting 
verifications  as  well  as  performance  verifica¬ 
tions.  For  example,  the  CPA  data  indices  are 
useful  for: 

o  insuring  that  ACWP,  BCWP,  AND  BCWS  are  less 
than  or  equal  to  budget  at  completion  (BAC) 
or  EAC, 

o  insuring  that  program  cost  data  for  the 
period  are  less  than  or  equal  to  cumulative 


data, 

o  insuring  that  cumulative  program  coat  data 
are  less  than  or  equal  to  EAC's, 
o  monitoring  cost  and  schedule  variances 
above  either  a  dollar  or  percentage  of 
cost  threshold  for  either  cumulative  or 
periodic  cost  account  data, 
o  monitoring  slippages  on  the  critical  path, 
o  flagging  cost  performance  anomalies, 
o  analyzing  contractor's  LRE  and  predicted 
EAC  values. 


Figure  3 
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Unfortunately,  the  predicted  EAC  values  and 
the  contract  condition  indicator  utilized 
in  the  CPA  model  depend  to  a  significant 
extent  on  ratios  which  contain  baseline 
costs.  Moreover,  two  of  the  time-weighted 
EAC  projections  require  six  and  eight  months 
of  data.  Consequently,  when  a  contract 
baseline  change  occurs,  the  EAC  projections 
are  confounded  and  an  erroneous  indication  of 
contract  status  is  generated. 

The  limitations  of  the  CPA  model  in  the  con¬ 
text  of  baseline  changes  motivated  consider¬ 
able  research  into  techniques  for  short-term 
risk  assessment  and  long-range  risk  manage¬ 
ment.  The  results  of  the  research  into 
short-term  techniques  are  embodied  in  the 
curve-fitting  module,  while  the  other  risk 
assessment  indicators  sre  oriented  toward 
longer  range  concerns.  Collectively,  these 
risk  assessment  indicators  compensate  for 
some  of  the  inadequacies  of  the  CPA  model 
when  contract  conditions  invalidate  the  CPA 
model  methodology. 

Curve  Fitting  Algorithm 

The  curve  fitting  algorithm  was  developed  in 
response  to  the  need  for  a  mechanism  to 
assess  short  term  cost  growth  which  is  not 
dependent  upon  cost  performance  ratios. 
Consequently,  the  algorithm  was  designed  to 
utilize  recent  ACWP  data  to  provide  a  clear 
and  useful  display  of  near  term  future  coat 
variability. 


The  approach  taken  to  the  quadratic  curve 
fitting  algorithm  consisted  of  the  projection 
of  ACWP  for  1,  2,  3,  and  4  months  into  the 
future.  Three  separate  projections  were 
prepared  for  each  month  based  on  the  contrac¬ 
tor's  ACWP  as  reported  in  the  previous  3,  4, 
and  5  Cost  Performance  Reports.  Three 
separate  projections  were  deemed  appropriate 
because  the  methodology  needed  to  be  aenaitivi 
to  relatively  long  term  indications  of  sta¬ 
bility  and  yet  be  responsive  to  short  term 
fluctuations  which  may  be  indicative  of 
rapidly  rising  costs. 

It  was  reasoned  that  the  more  months  upon 
which  the  projection  is  baaed,  the  more 
stable  the  projection  will  be,  i.e.,  the  less 
responsive  the  projection  will  be  to  singular 
cost  surges  or  plunges.  On  the  other  hand, 
cost  projections  based  on  fewer  months  of 
past  ACWP  data  are  more  sensitive  to  recent 
cost  fluctuations.  Hence,  the  dispersion  or 
envelope  of  cost  projections  derived  from 
various  time  horizons  is  an  indicator  of 
recent  cost  growth  variability,  in  addition 
to  providing  estimates  of  short  term  costs. 

An  example  of  the  type  of  output  which  the 
algorithm  provides  is  shown  in  Figure  4.  As 
shown,  the  series  of  projections  which  the 
algorithm  generates  form  an  envelope  which 
bounds  the  range  of  future  costs. 

Figure  4 
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The  generation  of  this  envelope  is  useful  for 
three  distinct  functions.  First,  as  a  measure 
of  relative  uncertainty,  an  analysis  of  the 
widths  of  the  envelopes  surrounding  the  pro¬ 
jections  of  individual  WBS  elements  can  assist 
the  PMO  in  identifying  particularly  volatile 
cost  elements  which  merit  closer  technical 
observation.  In  addition,  such  an  analysis 
would  also  necessarily  indicate  recent  cost 
growth  variability  within  the  UBS  element. 
Second,  the  curve  fitting  algorithm  provides 
useful  basis  for  comparison  with  the  contrac¬ 
tor's  baseline  plan.  Cost  projections  in 
excess  of  the  baseline  plan  should  provide  an 
early  warning  signal  to  the  PMO,  especially  1 
two  or  more  of  the  projections  are  above  the 
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baseline.  Third,  utilizing  the  outputs  of 
the  algorithm  retrospectively,  the  PMO  can 
assess  significant  deviations  between  ACWP 
and  the  previous  month's  projections.  If 
the  deviations  show  higher  than  projected 
costs,  the  causes  for  such  cost  growth  should 
be  examined.  Alternatively,  if  the  ACWP 
figures  fall  below  projections,  a  further 
comparison  with  the  baseline  should  be  made. 
ACWP  below  baseline  may  be  indicative  of 
either  a  schedule  slippage  or  an  extremely 
high  baseline  plan.  Also,  the  previous 
month's  projections  should  be  monitored  for 
changes  in  the  designators  of  the  lower  and 
upper  bounds.  In  general,  a  period  of 
steadily  increasing  ACWP  will  give  rise  to 
5-month  projections  forming  the  upper  bound 
and  3-month  projections  forming  the  lower 
bound.  However,  recent  significant  cost 
surges  will  create  a  situation  in  which  the 
5-month  and  3-month  projections  switch  tem¬ 
porarily.  This  switch  can  be  utilized  as  an 
early  warning  signal  by  the  PMO,  who  may 
alert  the  Technical  Office  to  the  cost  growth 
and  promptly  Implement  procedures  for  cost 
control. 

The  curve  fitting  algorithm  was  tested 
against  data  obtained  from  the  Teal  Ruby 
Program.  The  results  confirmed  that  the  size 
of  the  cost  prediction  envelope  varied  with 
cost  element  uncertainty.  Moreover,  it  was 
found  that  the  envelope  bounded  future  costa 
in  772  of  the  cases.  An  analysis  of  the 
wrong  predictions  indicated  that  a  substan¬ 
tial  majority  (812)  were  associated  with 
significant  changes  to  the  contract  baseline. 
Consequently,  in  the  absence  of  baseline 
changes,  the  curve  fitting  algorithm  was 
shown  to  bound  the  future  ACWP  in  952  of  the 
cases.  For  comparative  purposes,  a  second 
cost  envelope  was  constructed  by  taking  as 
bounds  the  contractor's  baseline  plus  or 
minus  102.  This  envelope  was  capable  of  pre¬ 
dicting  future  ACWP  in  only  612  of  the  cases. 

A  significant  result  of  this  analysis  was  the 
behavior  cf  the  algorithm  in  the  region 
surrounding  baseline  changes.  As  indicated 
earlier,  baseline  changes  have  historically 
rendered  the  projections  and  assessments 
resulting  from  the  CPA  model  erroneous  due  to 
the  confounding  effect  on  the  model's  regres¬ 
sions.  This  confounding  effect  lingers  for 
several  reporting  periods  after  the  baseline 
changes,  since  some  CPA  model  indicators 
utilize  up  to  eight  periods  of  historical 
data.  In  contrast,  the  curve  fitting 
algorithm  was  able  to  compensate  for  baseline 
changes  within  a  period  of  2-3  months. 
Consequently,  although  baseline  changes 
account  for  nearly  all  the  erroneous  predic¬ 
tions  of  the  curve  fitting  algorithm,  the 
algorithm  is  sufficiently  resilient  to  . 
recover  from  such  changes  within  a  relatively 
short  period  of  time.  , 


Finally,  it  should  be  noted  that  the  output 
of  curve  fitting  algorithm  cannot  be  analyzed 
without  regard  to  other  contract  indicators, 
such  as  the  narrative  portion  of  the  Cost 
Performance  Report  (CPR).  By  realizing  that 
a  baseline  change  is  Imminent,  the  PMO  can 
adjust  the  interpretation  of  the  algorithm's 
output  in  recognition  of  the  changing  base¬ 
line.  In  such  a  case,  less  emphasis  would  be 
placed  on  the  cost  envelope  per  se,  with  a 
corresponding  increase  in  emphasis  on  the 
retrospective  analyses  described  earlier. 

Rayleigh  Analysis 

The  Rayleigh  equation  describes  a  random  sta¬ 
tistical  process  where  many  complex  interac¬ 
tions  drive  the  unpredictability  of  the  pro¬ 
cess  outcome.  This  theory  has  been  used  to 
project  software  system  coat  based  upon  para¬ 
meters  of  development  time,  system  difficulty, 
and  number  of  source  statements.  It  has  also 
been  used  to  dynamically  track  manpower  and 
cash  flow  over  project  duration.  R&D  projects 
are  thought  to  exhibit  similar  patterns  of 
manpower  loading  and  interactions  as  those  of 
software  development,  where  frequently,  soft¬ 
ware  itself  is  an  Integral  part  of  R&D  pro¬ 
jects.  Preliminary  research  has  indicated 
that  a  potential  exists  for  the  application  of 
Rayleigh  theory  to  contract  costs. 

Each  cycle,  or  vcriable,  which  makes  up  part 
of  the  total  process  exhibit  a  pattern  of  a 
sharp  i  "  al  rise  in  manpower,  a  peak,  and 
then  gradua  tailing  off.  Figure  5  illus¬ 
trates  the  interactions  of  each  of  the  R&D 
Figur*  • 
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Ject  curve.  The  total  project  curve  tends  to 
demonstrate  the  Rayleigh  theory  also,  though 
not  as  pronounced  as  the  individual  cycles. 


The  mathematical  model  which  represents  this 
curve  snd  hss  been  successfully  used  for  both 
individual  cycles  and  overall  time  varying 
behavior  is  given  by: 


Y  -  k/td2  t 


"t2/2td2 


(1) 


where 
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Y  “  manpower  at  any  time  t 
K  «  area  under  the  curve  and  represents 
total  life  cycle  effort  in  man  years 
tj  ■  development  time  in  years  (time  of  peak, 
manpower) 

t  “  elapsed  time  since  initiation  of  project 

The  results  of  exercising  this  model  allow 
one  to  make  projections  of  manpower  and  time 
requirements  given  a  few  early  actual  data 
points.  These  projections  can  be  used  as  a 
basis  for  comparison  with  actual  project 
values  in  the  later  stages.  If  actual 
figures -exceed  the  established  bounds  sur¬ 
rounding  the  projection,  program  management 
may  be  alerted  that  some  corrective  action  is 
needed  to  bring  the  project  back  into  line. 

When  the  Rayleigh  theory  provides  a  good  ex¬ 
planation  for  varying  contract  costs,  the 
PMO  can  use  this  methodology  to  predict  total 
cost  as  well  as  next  period  costs  and  period 
of  greatest  cost  expenditure.  As  actual  cost 
data  becomes  available  for  each  period,  the 
model  can  be  updated  and  all  prior  data 
elements  used  to  predict  desired  parameters. 

A  close  watch  should  be  kept  on  previous  pre¬ 
dictions  versus  actual  costs  as  they  occur  so 
that  program  management  is  alerted  in  a 
timely  manner  of  a  potenti.'.l  problem. 

The  best  use  of  the  Rayleigh  model  is  as  a 
predictor  of  total  project  costs.  Applying 
the  Rayleigh  theory  to  Teal  Ruby  data  showed 
the  greatest  utility  of  Rayleigh  theory 
appears  to  be  in  projecting  cumulative  pro¬ 
ject  costs .  Figure  6  graphically  shows  the 
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the  project  cumulative  costs  versus  the 
cumulative  and  final  total  costa  predicted 
by  the  Rayleigh  curve.  The  graph  illustrates 
the  costs  at  the  end  of  quarter  9.  Nine 
quarters  of  data  were  input  to  the  model  and 
an  expected  totel  cost  of  51  million  dollars 
spent  by  the  expected  completion  at  quarter 
14  resulted.  This  compared  to  the  contrac¬ 


tor's  estimate  of  60.2  million  (BAC)  spent  b 
an  expected  completion  at  quarter  20.  The 
variance  in  cost  and  completion  time  between 
the  actual  and  predicted  curves  give  an  earl;, 
warning  signal  ot  potential  ^roile™*.  In 
the  Teal  Ruby  case,  there  were  in  fact  at 
least  two  baseline  changes  after  quarter  9. 
The  potential  for  the  increases  created  by 
the  baseline  change  are  evident  upon  examina- 
nation  of  these  graphs. 

The  Beta  Distribution  Model 

The  Beta  distribution  curve  has  been  success¬ 
fully  used  as  a  model  for  spreading  or  simul. 
ting  costs  across  time.  The  variables  in  th 
application  are  fractional  time  versus  frac¬ 
tional  cost  expended  or,  the  ratio  between 
resources  expended  and  time  consumed.  The 
Beta  curve  has  proved  useful  in  spreading  to¬ 
tal  program  costs  for  budgeting  and  scheduli 
purposes  as  well  as  a  powerful  tool  for  moni 
toring  on-going  projects  by  calculating 
expected  cost  at  completion  (EAC) .  This  in¬ 
formation  provides  managers  with  timely  feed¬ 
back  on  their  progress  in  meeting  cost  objec¬ 
tives  as  they  occur. 

The  cost  spreading  function  used  with  the  Be- 
model  was  developed  by  NASA  cost  analysts  at 
the  Johnson  Space  Center.  It  has  been  used 
extensively  by  NASA  to  project  EAC  at  succes¬ 
sive  time  periods  during  ongoing  projects. 
This  particular  model  gives  an  envelope  of 
cumulative  cost  curves,  as  shown  in  Figure  7, 

Figure  7 

BETA  DISTRIBUTION  ENVELOPE  OF 
CUMULATIVE  PROGRAM  COSTS 


that  can  be  expected.  The  model  best  fits, 
by  least  squares,  the  actual  project  data 
collected  to  dato  to  a  Beta  curve  within  thi 
envelope.  The  parameters  calculated  for 
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developing  the  curve  are  then  used  to  project 
an  EAC  given  the  project  data  to  this  point. 

The  moat  attractive  feature  of  this  model  is 
its  simplicity  of  use.  Data  inputs  are  pro¬ 
ject  start  date,  project  end  date,  current 
period  date  and  cumulative  cost  to  date.  Once 
input,  the  model  calculates  EAC.  This  EAC  can 
then  be  compared  against  the  contractor's  LEE 
and  against  other  estimates  of  EAC  from  other 
sources. 

The  output  from  this  model  is  valuable  for 
•  ••  two  reasons.  First,  it  provides  an  estimate 
of  total  cost  based  up.-n  actual  project 
duration  and  the  rate  of  cost  expended  to 
date.  This  can  be  compared  to  the  contrac¬ 
tor's  Latest  Revised  Estimate  (LRE)  which  is 
based  upon  the  same  length  of  project  time. 
Second,  it  provides  an  additional  estimate  of 
EAC  based  upon  different  parameters  than  the 
Rayleigh  model  uses.  Although  the  Rayleigh 
model  also  evaluates  rate  of  costs  expended, 
it  calculates  and  uses  its  own  expected 
project  duration  time.  Because  the  Beta 
model  projects  EAC  based  upon  different 
inputs,  it  provides  an  "independent"  assess¬ 
ment  of  EAC  for  comparative  uses.  This 
allows  the  LRE  to  be  assessed  from  two 
-.iCii  different  points  of  view. 


points  are  considerably  higher  than  LRE, 
there  is  an  indication  that  based  upon  the 
rate  of  early  expenditures,  the  project  will 
either  have  to  be  extended  or  cost  overruns 
will  be  large.  This  can  be  an  early  warning 
signal  to  expect  extended  project  duration, 
increased  LRE,  or  both. 

Parametric  Milestone  Analysis 

Parametric  milestone  analysis  refers  to  a 
procedure  whereby  ACWP  data  are  systematically 
compared  to  1)  projections  yielded  by  other 
techniques  in  the  risk  assessment  package,  snd 
2)  empirical  data  collected  on  other  high  risk 
R&D  programs.  The  purpose  of  the  parametric 
mileston  analysis  is  to  indicate  to  the  PMO 
how  the  progress  —  end  the  projected  progress 
—  of  the  contractor  compares  to  the  progress 
attained  by  other  contractors  conducting 
similar  R&D  programs.  The  progress  reported 
by  the  contractor  is  measured  with  respect  to 
the  achievement  of  key  project  milestones 
(e.g..  Critical  Design  Review,  Functional  Con¬ 
figuration  Audit,  etc.).  The  cost  projected 
to  be  incurred  by  the  contractor  at  the  mile¬ 
stone  as  a  percentage  of  total  contract  cost 
is  then  compared  to  the  percentage  .figure 
actually  incurred  on  contracts  of  a  similar 
nature. 


The  model  was  applied  to  the  Teal  Ruby  data 
base  and  produced  a  fit  of  the  data  to  the 
Beta  distribution.  This  model  is  sensitive 
to  project  duration  estimates  and  as  the 
length  of  the  project  is  revised,  the  esti¬ 
mates  of  EAC  become  more  reasonable.  Figure 
8  show  the  cumulative  Beta  curve  estimates  at 
Figure  8 

TEAL  RUBY  DATA  APPLIED  AGAINST  BETA 
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Parametric  milestone  analysis  is  not  intended 
to  provide  a  rigid  normative  model  of  how 
’  costs  necessarily  must  be  incurred,  but 
rather,  is  most  useful  as  an  early  warning 
indication  that  contractor  performance  may  be 
out  of  line  with  other  historical  experience. 
In  such  circumstances,  the  analysis  can  serve 
as  a  catalyst  to  the  closer  examination  of 
individual  cost  elements  which  lie  outside  the 
nominal  range. 

Figure  9 

PARAMETRIC  MILESTONE  ANALYSIS 
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the  midpoint  of  the  data  base.  Used  in 
conjunction  with  the  output  from  the  Rayleigh 
model,  the  Beta  distribution  model  provides  a 
different  assessment  of  total  expected  cost. 
The  output  also  provides  an  assessment  qf  the 
reasonableness  of  project  duration  time.  If 
values  of  EAC  projected  with  e^rly  data 


As  shown  in  Figure  9,  outputs  from  the  CPA 
model,  curve  fitting  algorithm,  and  Rayleigh 
analysis  model  are  used  as  inputs  to  the  para¬ 
metric  milestone  analysis,  in  sddltion  to  the 
contractor's  baseline  plan  (which,  of  course, 
sppears  in  the  curve  fitting  module  snd  is 
implicit  in  the  CPA  model).  If  in  sny  given 
month  no  milestones  appear  within  the  range  of 
the  curve  fitting  module,  then  the  output  from 
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that  module  Is  not  utilized  for  parametric 
analysis  In  that  month.  The  project 
these  sources  are  then  compared  to  empirical 
data  on  the  cumulative  percentage  expendi¬ 
tures  required  to  attain  the  corresponding 
milestones  on  similar  projects.  The  output 
of  the  parametric  milestone  analysis  is  a 
structured  listing  of  projections  from  each 
of  the  data  sources.  This  listing  also 
provides  an  assessment  of  the  cost  variance 
and  the  schedule  variance  predicted  by  each 
,of  the  four  methods.  The  ultimate  value  of 
the  report  lies  in  its  ability  to  provide  an 
indication  of  potential  problems  based  upon 
the  cost  and  schedule  variances  which  it 
computes . 

An  example  of  how  the  parametric  milestone 
analysis  may  be  applied  is  shown  in  Figure 
10.  Although  the  data  shown  were  obtained 

Figure  10 

PARAMETRIC  MILESTONE  ANALYSIS 
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from  the  contractor's  baseline  plan,  the  pro¬ 
cedures  for  analysis  of  actual  data  would  be 
identical.  In  addition  to  the  actual  cumula¬ 
tive  data,  parametric  milestone  analysis  con¬ 
tains  columns  for  estimates  derived  from 
short  term  CPA  model  forecasts,  curve  fitting 
projections  if  the  milestones  fall  within  the 
range  of  the  algorithm's  predictions,  and 
Rayleigh  theory  analysis  extrapolated  to  the 
individual  milestones. 

The  results  of  the  parametric  milestone 
analysis  showed  that  the  phase  costs  for 
software  were  remarkably  consistent  to  the 
empirical  data  base.  However,  when  compared 
to  the  Rayleigh  theory  estimates  for  software 
development,  the  technical  software  mile¬ 
stones  as  specif  iced  in  the  baseline  plan 
appeared  to  be  more  optimistic  than  the 
corresponding  milestones  proposed  for  the 
experiment  as  a  whole.  Moreover,  a  dispro¬ 
portionate  percentage  of  the  optimism  in  the 
proposed  software  plan  concerned  the  sched¬ 
uling  of  the  later  software  milestones.  The 
implication  of  this  parametric  milestone 
analysis,  when  viewed  in  conjunction  with 
the  results  of  the  Rayleigh  model  estimates, 
is  that  a  specific  WBS  element  could  be 
identified^  as  very  likely  to  cause  a 


slippage  in  the  baseline  plan.  In  addition, 
the  analysis  indicated  that  software  was 
likely  to  become  a  limiting  activity  on  the 
critical  path.  Based  upon  this  analysis,  the 
Technical  Office  was  able  to  focus  the  direc¬ 
tion  of  further  risk  management  activities 
and  Initiate  an  active  risk  reduction  program. 

Bayesian  Analysis 

The  purpose  of  Bayesian  analysis  is  to  provid 
the  PMO  with  information  on  the  uncertainty 
surrounding  the  estimated  costs  at  completion 
Bayesian  analysis  explicitly  recognizes  the 
uncertainty  in  cost  estimates,  and  rather  tha 
attempting  to  pinpoint  a  single  estimate  of 
cost  at  completion,  it  instead  characterizes 
the  range  of  possible  values  which  the  cost  a 
completion  may  include. 

The  Bayesian  analysis  algorithm  postulates  a 
probability  distribution  surrounding  initial 
cost  estimates.  The  characteristics  of  this 
probability  distribution  will  vary  depending 
on  the  nature  of  the  cost  element  under  con¬ 
sideration,  but  in  all  cases  may  be  defined  1 
terms  of  the  mean,  and  the  variance,  o'2 
Utilizing  the  properties  of  Bayesian  statis¬ 
tics,  the  empirical  data  for  the  cost  element 
as  reported  by  the  contractor  on  the  Cost 
Performance  Report  (CPR)  are  used  to  update 
the  characteristics  of  the  probability  distri 
butlon.  P'3ed  upon  this  new  information  as 
reported  in  the  CPR,  the  parameters  of  the 
distribution  are  updated.  Including  both  the 
mean  and  the  variance. 

The  outputs  of  this  analysis  are  updates  of 
the  initially  postulated  distributions.  Thes 
updates  permit  three  distinct  analyses  to  be 
performed.  First,  a  new  mean  for  the  distri¬ 
bution  will  be  provided.  Under  commonly  used 
distributions,  the  mean  also  corresponds  to 
the  value  with  the  highest  probabililty  of 
occurrence.  Therefore,  the  new  mean  may  be 
construed  as  indicating  a  new  estimate  at 
completion . 

Second,  the  new  mean  will  also  have  a  proba¬ 
bility  distribution  surrounding  it.  Utilizir 
this  distribution,  the  Bayesian  analysis 
algorithm  establishes  confidence  intervals  or 
the  mean.  The  higher  of  these  confidence 
intervals  less  the  value  of  the  mean  may  be 
interpreted  as  the  level  of  contingency  re¬ 
serve  allocation  necessary  to  achieve  a  give 
confidence  level  on  the  estimated  cost  at  co 
pletion.  The  PMO  may  then  alter  the  width  o 
the  confidence  interval  to  decrease  (or 
increase)  the  amount  of  contingency  reserves 
required,  and  incur  a  corresponding  decrease 
(or  increase)  in  the  confidence  level  of  the 
estimate  at  completion.  In  this  manner,  the 
PMO  may  trade  off  risk  reserves  versus  the 
increased  uncertainty  in  remaining  within  th 
budget  at  completion. 
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Finally,  the  new  probability  diatrlbution  may 
be  compared  with  the  previoua  diatribution. 
The  essence  of  this  analysis  is  illustrated 
in  Figure  11.  This  comparison  ia  useful  for 
three  purposes.  First,  the  PMO  may  compare 
how  the  estimated  mean  has  changed  from  month 
to  month.  The  magnitude  of  this  change  will 
Indicate  the  level  of  Increase  in  expected 
'  cost  at  completion  from  the  previous  month. 

Second,  the  PMO  may  analyze  the  change  in  the 
variance  from  the  previoua  month.  An  in- 
creaae  in  the  variance  will  Indicate  greater 
'JlJ',  uncertainty  in  the  cost  estimates.  Converse¬ 
ly,  a  decrease  in  variance  will  indicate  that 
the  estimate  at  completion  is  becoming 
firmer.  Such  a  development  might  occur  near 
project  completion.  The  third  purpose  this 
comparison  serves  is  to  show  the  new  probar 
blllty  of  attaining  an  actual  cost  at  comple¬ 
tion  which  is  less  than  or  equal  to  the 
previous  month's  estimated  cost  at  comple¬ 
tion.  The  probability  is  represented  by  the 
shaded  area  in  Figure  11.  Such  an  analysis 
shows  that,  as  new  estimates  at  completion 
are  developed,  the  probability  of  achieving 
-  i""  the  former  estimates  at  completion 

diminishes.  This  in  turn  may  indicate  that 
previously  allocated  management  reserves  are 
not  sufficient  and  that  a  baseline  change  is 
rjv*  necessary. 

Figure  11 


As  emphasized  previously  in  this  report,  the 
techniques  presented  for  DARPA  program  risk 
assessment  are  oriented  toward  the  special 
needs  of  the  PMO  to  identify,  to  analyze,  and 
ultimately  to  manage  risk.  The  techniques 
were  designed  to  be  used  as  tools  in  conjunc¬ 
tion  with  other  information  available  to  the 
PMO.  Taken  individually,  none  is  an  infal¬ 
lible  Indicator  of  program  cost  and  cost 
growth  risk.  Taken  collectively,  they  yield 
a  reasonably  comprehensive  assessment  of 
program  risk  from  numerous  different  perspec- 
tives. 

The  applicability  of  each  of  ^he  program  risk 


assessment  techniques  is  summarized  in  Figure 
12.  As  shown  in  this  figure,  the  techniques 

Figure  12 
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have  two  fundamental  purposes.  First,  they  are 
Intended  to  serve  as  early  warning  signals  to 
the  PMO  to  alert  management  in  a  timely  manner 
to  the  possibility  of  unexpected  cost  growth. 
Second,  the  techniques  are  Intended  to  provide 
an  estimate  or  an  indication  of  the  likely 
magnitude  of  a  cost  overrun  or  schedule  slip¬ 
page.  As  a  result,  the  analytical  framework 
recommended  for  DARPA  program  risk  assessment 
consists  of  three  principal  elements  which 
focus  on  short  term,  mid  term,  and  long  term 
concerns.  Within  each  of  these  elements,  the 
outputs  of  two  or  more  of  these  techniques  may 
be  synergistically  combined. 

Short  Term  Analysis 

Short  term  analysis  can  be  accomplished 
through  use  of  the  CPA  model  and  the  curve- 
fitting  algorithm,  giving  cost  prediction  and 
early  warning  signals.  These  outputs  permit 
the  following  analyses  to  be  performed: 

o  Evaluation  of  earned  value  data.  CPA  model 
output  contains  ratios  that  permit  the 
evaluation  of  recent  past  contractor  perfor¬ 
mance  in  terms  of  ACWP  vs.  BCWP  (cost  over¬ 
runs)  and  in  terms  of  BCWP  vs.  BCWS 
(schedule  slippage).  Deviations  greater 
than  3Z  generate  early  warning  messages, 
o  Cost  prediction.  Curve  fitting  algorithm 
projects  near-term  future  cost  and  enables 
comparison  of  projection  to  contractor's 
baseline.  Significant  deviations  generate 
early  warning  message, 
o  Cost  analysis.  Curve  fitting  algorithm 
enables  a  "backward  look"  at  the  actual 
cost  data  to  permit  a  comparison  to  previous 
predictions.  Overruns  or  underruns  command 
management  attention  to  control  costa  (or 
increase  the  baseline)  in  the  case  of  the 
former  or  to  examine  data  for  schedule  slip¬ 
pages  in  the  case  of  the  latter.  This 
analysis  should  be  done  in  conjunction  with 
CPA  model  indicators. 
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element,  permits  the  Identification  of 
those  cost  elements  with  the  largest  pro¬ 
jected  cost  envelope.  This  is  in  turn 
indicative  of  those  cost  elements  which 
have  shown  the  greatest  short  term  cost 
variability  in  recent  months. 

Mid  Term  Analysis 

In  the  mid  term,  the  risk  assessment  indi¬ 
cators  focus  less  on  the  prediction  of  costs 
and  more  on  the  assessment  of  the  reasonable¬ 
ness  of  the  expenditure  patterns  vis-a-vis 
theoretical  and  empirical  models.  From  the 
theoretical  perspective,  comparisons  with  the 
Rayleigh  theory  of  expenditure  patterns  ars 
possible  for  any  reporting  period  in  the  life 
of  the  project.  At  certain  other  points,  the 
contribution  of  Beta  theory  analysis  is 
applicable.  From  the  empirical  perspective, 
parametric  milestone  analysis  can  provide  a 
benchmark  bas  -d  on  historical  experience 
against  which  -rogress  may  be  measured.  In  a 
limited  number  of  cases,  the  curve  fitting 
module  will  have  applicability  through  its 
impact  on  the  parametric  milestone  analysis 
output.  In  summary,  the  mid  term  risk  indi¬ 
cators  have  applicability  as  follows: 

o  Cumulative  coat  analysis.  A  project’s 
cumulative  costs  versus  milestones  are 
compared  to  historical  experience  to  indi¬ 
cate  significant  deviations.  Reasons  for 
large  deviations  (either  positive  or 
negative)  should  be  explained  in  detail  to 
the  PMO  as  they  may  indicate  either  a  com¬ 
pressed  time  frame  or  the  imminent  need  for 
baseline  revisions. 

o  Expenditure  pattern  analysis.  Expenditure 
pattern  analysis  refers  to  analyses  of 
long  term  trends  in  expenditures  as  opposed 
to  short  term  expenditures  which  may  fluc¬ 
tuate  with  the  business  cycle.  Basically, 
expenditure  pattern  analysis  entails  com¬ 
parison  to  Rayleigh  theory  predictions  for 
the  pattern  of  expenditures  and  the  Beta 
theory  estimates.  In  addition,  this 
analysis  Includes  the  examination  of  costs 
at  the  UBS  level  and  indicates  which  cost 
elements  are  experiencing  recent  changes 
from  established  patterns. 

Long  Term  Analysis 

Long  term  analysis  Includes  all  of  the  esti¬ 
mates  at  completion  as  determined  through  the 
individual  techniques  plus  confidence  inter¬ 
vals.  The  following  analyses  may  be  performed 
using  the  risk  assessment  Indicators: 

o  Estimate  at  Completion  (EAC).  The  CPA  model 
provides  seven  estimates  at  completion 
including  the  contractor’s  latest  revised 
estimate  (LRE).  Bayesian  analysis,  while 
constrained  by  assumptions,  does  provide  a 
maximum  likelihood  estimate  of  EAC  based  upon 


a  contractor's  monthly  reports  of  ACWP.  In 
addition,  the  Rayleigh  model  provides  an  EAC 
based  upon  the  development  times  of  the 
program,  and  Beta  theory  provides  an  EAC 
based  upon  the  starting  date,  ending  date, 
and  cumulative  cost  to  date.  These  estimates 
may  be  compared  and  contrasted  to  yield  an 
assessment  of  long  term  cost  growth  risk, 
o  Confidence  limits.  Bayesian  analysis  also 
provides  a  confidence  interval  for  the  esti¬ 
mate  at  completion.  The  width  of  this 
Interval  indicates  the  confidence  one  may 
have  in  the  estimate,  or  conversely,  the 
relative  risk  that  the  actual  cost  will  not 
be  near  the  EAC. 

o  Probability  estimates.  These  estimates  are 
derived  from  the  Bayesian  analysis,  and 
permit  the  PMO  to  assign  a  probability  to 
achieving  a  final  cost  less  than  or  equal  to 
a  given  value  (e.g.,  contractor's  LRE,  con¬ 
tractor's  LRE  +  10Z,  etc.).  Conversely, 
these  estimates  also  permit  the  PMO  to  assess 
the  amount  of  contingency  reserves  necessary 
to  bound  a  confidence  interval  of  some 
desired  percentage.-Oe.g. ,  $3  million  reserves 
necessary  to  achieve  90Z  confidenca  inter¬ 
val). 

SUMMARY  AMD  CONCLUSIONS 

This  paper  has  described  an  innovative  approach 
to  risk  assessment  for  use  in  the  DARPA  Program 
Management  Office.  The  approach  focuses  on  the 
prediction  of  cost3  in  the  short  run  and  the 
indication  of  risk  over  longer  time  horizons. 

It  employs  a  quadratic  curve-fitting  algorithm 
to  estimate  short  term  cost  fluctuations,  and 
it  uses  theoretical  and  empirical  cost  models 
both  to  estimate  the  cost  at  completion  and  as 
well  as  to  gauge  the  reasonableness  of  the 
expenditures  to  date.  The  components  of  the 
approach  are  intended  to  be  utilized  collec¬ 
tively  to  yield  a  balanced  appraisal  of  cost 
and  schedule  risk  of  a  program  over  the 
remaining  phases  of  the  R&D  cycle. 

In  summary,  this  report  has  documented  the 
initial  steps  which  have  been  taken  to  pro¬ 
vide  DARPA  PUD  with  an  augmented  analytical 
risk  assessment  capability.  The  basic  risk 
assessment  tools  have  been  developed  and  are 
now  available  for  use  by  the  PMO.  The  adop¬ 
tion  of  the  methodology  will  assist  DARPA  in 
assessing  the  cost  and  schedule  risk  of  high 
risk  R&D  programs,  and  will  provide  the 
foundation  for  future  PMO  activities  in  risk 
management  and  control. 


Reference:  “Analysis  of  Quantitative  Techni¬ 
ques  for  DARPA  Program  Risk  Management”. 
Interim  Report  under  Contract  No.  MDA903-82-C- 
0139,  Meridian  Corporation.  May  19,  1983. 
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ABSTRACT 


Brain  research  has  grown  into  an  acceptable 
area  for  management  awareness.  Learning  facts 
about  the  brain  hemispheres  has  yielded  power¬ 
ful  tools  for  decisionmakers,  allowing  them 
to  become  literate  about  brain  functions,  to 
utilize  the^fact^fthat  the  two  sides  of  the 
brain  are  entirely  independent  with  regard  to 
learning  and  retention.  A  new  paradigm  incor¬ 
porating  consciousness  technology  may  better 
serve  the  decisionmaking  processes  of  our 
rapidly-changing  society.  , 


INTRODUCTION 


The  purpose  of  this  paper  is  to  put  the  emerg- 
^  ing  consciousness  technologies  into  a  manage- 
:  ;■  rial  perspective.  The  on-going  brain  research 
’  which  has  resulted  in  three  Nobel  Prizes  in 
1981  to  American-based  researchers,  focused  on 
specialized  higher  split-brain  functions. 


The  first  part  of  this  paper  explains  the 
underlying  theory  of  split-brain  research, 
indicating  the  more  significant  developments 
for  the  mechanisms  of  decisionmaking.  The 
second  part  reviews  significant  implications 
the  research  contributing  to  a  new  paradigm 
for  this  current  era  of  profound  change. 


of 


THE  TWO  MODES  OF  DECISIONMAKING 


Decisionmakers  have  been  aware  that  there  are 
many  different  paths  to  problem  solving.  When 
queried  about  the  mental  process  by  which  the 
"right"  answer  was  selected,  the  decisionmaker 
cannot  track  the  logical  process.  Darwin, 
Copernicus,  and  Newton  are  examples  of  the 
revolutionary  scientific  work  that  involves 
shifting  to  a  new  paradigm. 


Several  erroneous  assumptions  in  Nineteenth 
Century  science  have  oriented  our  cultural 
philosophy  in  the  direction  of  rational  mechan¬ 
ical  thinking: 


•  All  known  forms  of  energy  have  been 
discovered. 


•  Our  ordinary  consciousness  of  time/ 

space  is  necessarily  the  correct 

view. 

•  Men  live  as  separate  and  autonomous 

individuals . 

•  The  existence  of  the  physical  body  is 

the  essence  of  man. 

Concentrating  on  the  verbal  (left)  side  of  our 
brains  we  produced  things  we  know  how  to  anal- 
ze  and  measure.  Our  mute  (right)  brain  remain¬ 
ed  uncharted.  Current  research  is  directed  to 
discovering  how  the  right  hemisphere  thinks  and 
how  it  might  be  educated,  and  more  important 
what  its  contribution  is  in  the  complex  art  of 
creativity. 

Split-Brain  Research 

It  is  well  known  that  the  left  hemisphere  of 
the  brain  (the  "dominant"  or  "major"  hemi¬ 
sphere)  controls  movements  on  the  body's  right 
side,  and  the  right  hemisphere  (the  "subordi¬ 
nate"  or  "minor")  controls  movements  on  the 
left. 

Other  research  indicates  that  the  left  hemi¬ 
sphere  also  contains  the  logical  thinking  pro¬ 
cesses  for  most  people.  These  processes  are 
mainly  linear  —  processing  information  sequen¬ 
tially,  one  bit  after  another,  in  orderly 
analytical  fashion  —  line  mathematics  or  lang¬ 
uage.  The  right  hemisphere  in  contrast,  is 
specialized  for  simultaneous  processing;  i.e., 
it  functions  in  a  more  holistic,  relational 
way  —  like  comprehension  of  visual  images. 

The  Dichotomy,  quoted  in  Linstone  and  Simmonds' 
book  on  Futures  Research:  New  Directions  [9], 
shows  a  schematic  comparison  of  the  split-brain 
halves  and  their  biological-behavioral  patterns 

TABLE  1.  The  Dichotomy 


Left-Brain  Hemisphere 

Sequential  thinking 

Reductionist 

Analytic 

Well-structured  problems 
Problem  solvers 
Sensation  (Jung) 

Perceptual,  external  experience 
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Right-Brain  Hemisphere 

Spatial  thinking 

Holistic 

Synthetic 

Ill-structured  problems 
Problem  formulators 
Intuition  (Jung) 

Conceptual,  internal  experience 

The  human  being  can  be  thought  of  as  an  organ¬ 
ism  composed  of  components  having  both  psy¬ 
chological  ("receptive  mode"  to  receive  the 
environment)  and  biological  dimensions  ("action 
mode"  to  manipulate  the  environment).  In  his 
Harvard  Business  Review  article,  Henry  Minzberg 
[13]  stated 

. . .  scientists  have  further  found  that 
some  common  human  tasks,  activate  one  side 
of  the  brain  while  leaving  the  other 
largely  at  rest.  For  example,  a  person's 
learning  of  a  mathematical  proof  might 
evoke  activity  in  the  left  hemisphere  of 
his  brain,  while  his  conceiving  a  piece 
of  sculpture  or  assessing  a  political 
opponent  might  evoke  activity  in  his 
right. 

Robert  Ornstein  [15]  points  out  that  the 
"esoteric  psychologies"  of  the  East  (Zen,  Yoga, 
Sufi,  etc.)  have  focused  on  right-hemispheric 
consciousness,  while  Western  psychology  has 
been  almost  exclusively  concerned  with  left- 
hemispheric  consciousness,  with  logical, 
rational  thought. 

Inside  each  of  our  skulls,  we  have  a  double 
brain  with  two  ways  of  knowing;  each  half  with 
its  own  way  of  perceiving  external  reality. 

Each  of  us  has  two  minds,  two  consciousnesses, 
mediated  and  integrated  by  the  connecting 
cable  of  nerve  fibres  (corpus  callosum)  between 
the  hemispheres.  The  two  hemispheres  can  work 
together  in  a  number  of  ways.  Each  half  can 
cooperate  with  the  other  by  contributing  its 
special  abilities,  and  taking  on  the  peculiar 
part  of  the  task  that  is  suited  to  its  mode  of 
information  processing.  At  other  times,  the 
hemispheres  can  work  singly;  with  one  half 
"on",  the  other  half  more  or  less  "off".  The 
hemispheres  may  also  conflict;  one  half 
attempting  to  do  what  the  other  half  "knows" 
it  can  do  better.  It  may  even  be  that  each 
h=^isphere  has  some  way  of  keeping  knowledge 
from  the  other  hemisphere'. 

As  each  of  r.ne  hemispheres  gather  in  the  same 
sensory  information,  each  half  may  handle  the 
information  differently:  the  task  may  be 
divided  between  the  hemispheres,  each  handling 
the  part  suited  to  its  style.  Or  one  hemi¬ 
sphere,  often  the  dominant  left  will  "take 
over"  and  inhibit  the1  o'her  half.  The  left 
hemisphere  analyzes,  abstracts,  counts,  marks 
time,  plans*step-by-step  procedures,  verbalizes. 


makes  rational  statements  based  upon  logic. 

On  the  other  hand,  the  right  hemisphere  mode 
is  a  second  way  of  knowing.  We  "see"  things 
in  this  mode  that  may  be  imaginary,  or  recall 
events  that  may  be  real.  We  see  how  things 
exist  in  space,  and  just  how  parts  go  together 
to  make  up  the  totality,  the  gestalt.  Using 
the  right  hemisphere,  we  understand  metaphors, 
anagrams,  dreams;  we  create  new  combinations 
of  "Eureka"-type  ideas,  we  use  intuitions  and 
have  leaps  of  insight  and  "peak"  experiences. 
Most  of  our  training  —  home,  school,  business 
social  —  has  been  designed  to  cultivate  and 
prize  the  verbal,  rational,  on-time,  left-half 
—  while  virtually  neglecting  right-half  pro¬ 
cessing  . 

George  M.  Prince  of  Synectics,  Inc.,  saw  this 
"split-brain"  discovery  as  the  key  to  unlockin 
the  creative  process.  Reasoning  that  the  key 
to  creativity  lay  in  suppressing  the  left 
(or  logical)  hemisphere,  he  encouraged  his 
clients  to  become  comfortable  with  their  right 
brains,  the  so-called  "storehouse  of  ideas". 

In  his  "encounter"  sessions  he  came  up  with 
techniques  like  that  of  Goal-Wishes  —  in  whicl 
an  aspiring  inventor  would  fantasize  about  how 
his  problem  might  be  solved  if  there  were  no 
fiscal  or  technical  constraints.  From  this 
list  the  client  was  instructed  to  select  such 
an  absurd  method  of  achieving  his  goal  that  he 
would  fear  dismissal  if  he  seriously  proposed 
it  at  work.  This  "get-fired  solution"  was  one 
that  could,  in  many  cases,  be  refined  into  an 
imaginative  yet  workable  answer. 

In  attempting  to  repress  the  left-brain,  Dudle; 
Lynch  used  such  techniques  as  playing  soft, 
rhythmic  music  while  gazing  at  a  glass  coffee 
pot  onto  which  a  series  of  colored  lights  are 
projected.  Another  area  of  research  has  been 
with  the  use  of  drugs  designed  to  chemically 
suppress  the  working  of  the  left  hemisphere; 
on  the  theory  that  with  that  side  out  of  the 
way,  the  thoughts  of  the  right-brain  will  come 
forth.  Other  psychologists  are  convinced  that 
the  key  to  understanding  creativity  in  decisio 
making  can  be  found  through  the  techniques  of 
brain-scan  and  biofeedback.  The  Greens,  at 
the  Menninger  Foundation,  using  biofeedback 
have  been  able  to  achieve  and  prolong  the  thet 
state  (and  stimulate  creativity),  which  occurs 
in  the  immediate  pre-  and  post-sleep  stages. 
Professor  Eugene  Gendlin,  University  of  Chicag 
has  developed  a  technique  of  "focusing",  in 
which,  he  claims,  a  person  can  reach  theta  st 
and  remain  there  indefinitely  through  a  proce 
akin  to  self-hypnosis,  where  the  subject  make 
a  connection  between  the  brain  halves. 

It  seems  that  the  nature  of  the  task  at  hand 
may  induce  a  particular  hemisphere  to  "accept 
and/or  "control"  the  job,  while  repressing  th 
activity  of  the  other  hemisphere.  "Scientist 
postulate,"  stated  Betty  Edwards  in  her  Drawi 
on  the  Right  Side  of  the  Brain  [5]  "that  the 
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hemispheres  either  alternate  being  'on'  -  an 

'on'  condition  in  one  hemisphere  causing  ah 

'off'  condition  in  the  other  -  or  they  are 

both  'on' ,  but  with  one  hemisphere  controlling 
the  action  (the  overt  behavior)."  What  factors 
determine  which  hemisphere  will  be  "on"  and/or 
controlling?  She  concludes  that  scientists 
believe  that  the  control  question  can  be 
decided  mainly  in  two  ways: 

One  way  is  speed:  which  hemisphere  gets 
to  the  job  the.  quickest?  A  second  is 
motivation:  which  hemisphere  cares  most 
or  likes  the  task  the  best?  And  converse¬ 
ly:  which  hemisphere  cares  least  and 
likes  the  job  least?  Since  drawing  a 
perceived  form  is  largely  a  right-brain 
function,  we  must  keep  the  left-brain 
out  of  it.  Our  problem  is  that  the  left- 
brain  is  dominant  and  speedy  and  is  very 
prone  to  rush  in  with  words  and  symbols, 
even  taking  over  jobs  which  it  is  not 
good  at.  The  split-brain  studies  indi¬ 
cated  that  the  left-brain  likes  to  be 
boss;  so  to  speak,  and  prefers  not  to 
relinquish  tasks  to  its  "dumb"  partner 
unless  it  really  dislikes  the  job  — 
either  because  the  job  takes  too  much 
time,  is  too  detailed  or  slow  or  because 
the  left-brain  is  simply  unable  to  accom¬ 
plish  the  task. 

Figure  1,  taken  from  Bar tf eld,  Selman  and 
Selman  [2]  illustrates  a  suggested  decision¬ 
making  matrix  for  forecasting  purposes.  In 
cases  where  both  brain-sides  are  in  agreement, 
there  is  no  problem.  In  case  of  disagreement, 
recent  past  personal  experience  will  suggest 
the  acceptable  course  of  action;  if  not,  or  if 
experience  is  insufficient,  the  more  conserva¬ 
tive  "Don'*,  do  it"  prevails. 


How  does  one  use  "past  experience"  pramatical- 
ly?  By  comparing  the  positive  experience 
gained  in  similar  circumstances  when  following 
intuition  or  hunches  (like  betting  on  numbers), 
versus  positive  experience  gained  by  following 
an  analytical  approach  (like  betting  on  prob¬ 
abilities),  the  decisionmaker  can  select  that 
course  of  action  proven  most  profitable  for  him 
in  the  long  run. 

David  Loye  [11]  believes  that  the  study  of  the 
functions  of  the  forebrain,  the  so-called 
"frontal  lobes",  has  been  almost  as  neglected 
as  the  study  of  the  right-brain  functions. 

Forebrain  functions  are  of  three  main  types: 

•  Foresight,  including  detection  of 

change  patterns. 

•  Systems  thinking  —  the  ability  to  cog¬ 

nize  in  wholes. 

•  Self-regulation  —  the  ability  to  con¬ 

trol  behavior  that  shapes  one's  own 
future  by  interposing  something  that 
governs  behavior,  something  that 
says:  "Yes,  I  will  do  this",  or 
"No,  I  won't  do  that." 

To  the  above  forebrain  powers,  psychobiologist 
David  Goodman  adds  holos,  the  social  sense,  and 
prognos,  the  sense  of  the  future. 

After  identifying  the  three  separate  mind 
functions  as  concepts,  the  next  question  is: 

How  do  they  work  together?  David  Loye  in  his 
article  "The  Forecasting  Mind",  in  The  Futurist, 
speculates  on  the  working  together  of  the 
Right-,  Left-,  and  Forebrain,  and  how  they 
relate: 

Think  of  the  forebrain  as  the  mind's  man¬ 
ager  through  whom  all  questions  involving 
decisionmaking  must  pass.  Under  fore¬ 
brain  monitoring  and  guidance,  the  left 
brain,  primarily  through  consciousness, 
processes  the  information  received  from 

the  phenomenal  world  -  or  the  world  of 

"appearances"  -  much  as  a  computer  with 

pattern  recognition  equipment  would.  It 
scans  the  data;  if  it  then  detects  any¬ 
thing  with  which  it  is  programmed,  it  will 
predict  future  events. 

Also  with  forebrain  monitoring,  the  right 
brain,  primarily  through  unconsciousness, 
processes  information  from  ...  the 
noumenal  as  well  as  the  phenomenal  world. 

In  other  words,  the  right  brain  receives 
info-mation  whose  "reality"  is  unlike  that 
of  the  phenomenal  world  . . . 

From  these  two  sources,  information  bearing  on 
the  future  flows  into  the  forebrain.  Here, 
the  forebrain  must  decide  what  is  sense,  and 


FIGURE  1. 


Decision  hfctrix 
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what  is  nonsense,  and  what  should  be  done  about 

it  -  what  actions  to  take,  or  what  actions 

not  to  take.  David  Loye  in  his  pioneering 
work,  The  Knowable  Future  [10],  uses  the  fol¬ 
lowing  example  to  illustrate  the  integration 
of  the  left-,  right-,  and  forebrain. 

It  seems  evident,  then,  that  our  capacity 
for  foreknowledge  derives  from  pooling  and 
processing  by  the  forebrain  of  the  ration¬ 
al  left-brain  information  and  intuitive 
right-brain  information.  In  other  words, 
it  is  as  if  the  forebrain  acts  like  a 
field  general  to  whom  two  lieutenants 
bring  in  reports  in  the  heat  of  battle. 

One  lieutenant,  representing  left  brain 
rationality,  is  spotlessly  uniformed, 
clicks  his  heels  and  rattles  off  his 
information  like  a  computer.  The  other, 
representing  right  brain  intuition, 
silently  slips  in  clad  in  his  spy's 
gypsy  disguide  and  mumbles  and  rambles 
through  his  account.  The  forebrain  as 
field  general  then  sifts  the  perceptions 
of  both  lieutenants  and  decides  what  is 
the  likely  shape  of  the  future  course  of 
battle,  and  what  is  to  be  done  about  it. 

William  J.J.  Gordon  [7],  author  of  Synectics , 
contends  that  the  left,  or  logical,  part  of 
the  brain  is  more  involved  in  the  creative  pro¬ 
cess  than  many  other  researchers  believe.  In 
Gordon's  view,  the  formation  of  creative  ideas 
is  actually  an  oscillation  between  the  right 
and  left  hemispheres,  with  the  right  hemisphere 
continuously  making  free  associations  and  the 
left  performing  rapid  judgments  on  them  and 
sending  them  back  to  the  right  brain  for  more 
work.  Niles  Howard  [8],  in  his  "Business 
Probes  the  Creative  Spark",  in  Dun's  Review 
concludes  that: 

In  creativity  tests  on  individuals  of  all 
ages,  creativity  scores  invariably  drop 
about  90Z,  between  ages  five  and  seven, 
and  by  age  forty,  an  individual  is  only 
about  2Z  as  creative  as  he  was  at  age 
five.  This  suggests  to  many  psychologists 
that  the  almost  total  emphasis  on  logical 
thought  in  education  may  effectively  sup¬ 
press  creativity. 

The  expectation  of  further  "split-brain" 
research  is  that  what  has  been  trained  out  can 
be  trained  back  in.  A  creativity  test  designed 
by  Eugene  Raudsepp  (Princeton  Creative  Research, 
Inc.) is  included  as  Appendix  A,  How  Creative 
Are  You? 


UNCERTAINTY 

The  work  "futurology"  was  coined  to  describe 
the  scientific  study  of  the  future.  Before 
then,  images  of  the  future  include  frequently 


speculation,  guesswork  and  mysticism.  Predic¬ 
tion  is  a  revelation  of  what  can  happen,  given 
certain  conditions  and  occurrences.  Forecast¬ 
ing  allows  influencing  outcomes,  by  changing 
and  impacting  one  or  more  variables  involved. 
It  permits  society  to  preview  and  discuss  pos¬ 
sible  outcome  options  and  to  select  alterna¬ 
tives.  Newly  emerging  methodologies  for 
Decision  Risk  Research  included  approaches 
remote  from  conventional  Western  epistemology 
to  problem-solving,  which  was  primarily  built 
on  the  notions  of  the  "hard"  sciences  of 
replicable  experiments,  and  "objective"  truth. 
Looking  now  towards  a  complex  and  fuzzy  future 
with  uncertainties,  discontinuities,  and  un¬ 
knowns,  surprise-free  extrapolations  of  the 
past  and  present  become  rather  limited  and 
restricted  tools  for  forecasting.  There  is  no 
fail-safe,  risk-free  future.  Viewed  in  game- 
theoretic  concepts,  pre- industrial  society 
was  a  zero-sum  game  against  nature;  industrial 
society,  a  game  against  "rational  man",  and 
post-industrial  society,  and  n-person  game. 
Daniel  Bell  [  3]  suggested  coming  changes  in 
sociological  terms.  He  notes: 

Yet,  in  an  intangible  way,  there  may  be 
more  —  a  change  in  consciousness  and 
cosmology,  the  dark  tinge  of  which  was 
always  been  present  at  the  edges  of  man's 
conception  of  himself  and  the  world,  whic 
now  moves  to  the  phenomenological  center. 

H.A.  Linstone  and  W.H.C.  Simmonds  [9]  suggests 
specifically  a  fundamentally  different  approac 
to  analyzing  future  problems  than  the  more 
common  emphasis  on  more  numbers,  more  causal 
related  techniques: 

"...  move  beyond  the  objective,  analytic, 
reductionist,  number-oriented,  optimizing 
and  fall-safe  approaches  to  future  probli 
and  learn  to  think  with  equal  fluency  in 
more  subjective,  synthesizing,  holistic, 
qualitative,  option-increasing,  and  safe- 
fail  ways." 

Using  split-brain  concepts,  David  Loye  [10] 
accepted  the  validity  of  qualitative  informa¬ 
tion  and  the  use  of  judgmental  or  value-based 
decisionmaking  for  forecasting  processes.  Th 
is  can  be  inferred  that  less  analytical  methc 
may  have  a  significant  place  in  forecasting 
processes . 

In  order  to  assist  this  process,  a  group  of  ] 
of  the  most  common  forecasting  techniques  li 
by  Mitchell  [11]  was  reclassified  by  Bartfel 
Selman  and  Selman  [2]  into  three  groups  acco: 
ing  to  brain  function:  left-,  right-,  and 
mixed,  ,.nd  are  listed  in  the  Reference.  Per 
of  the  list  clearly  indicates  the  prepondera 
of  left-brain  based  methodologies  which  woul 
explain  some  of  the  dissatisfaction  voiced  b 
Armstrong  [1]  and  others  with  the  poor  recor 
of  decisionmakers  and  forecasters  in  our 


272 


fast-changing  environment. 


Another  emerging  technique  which  may  become  in 
the  future  increasingly  important  is  "meta¬ 
modeling"  . 

"Metamodeling"  can  be  described  as  a  formal 
framework  for  future  modeling,  which  utilizes 
both  human  subjective  assessments  and  conven¬ 
tional  mathematic  models  as  in  synergistic 
Decision  Support  Systems.  Future  modeling 
of  such  complex  areas  as  weapons  acquisition 
and  environmental  systems  can  be  approached  by 
the  use  of  fuzzy  variables  and  possibility 
theory  in  the  creation  of  future  scenarios. 
This  holistic  framework  attempts  to  combine 
human  judgment  into  the  structured  modeling 
process,  according  to  J.  Fiksel  [6]. 


SUMMARY 

It  is  clear  today,  looking  at  the  unsettled 
state  of  the  world,  and  the  quantum  changes 
taking  place  continually,  that  the  customary 
decisionmaking  tools  are  not  adequate  to  per¬ 
mit  timely  predictions.  In  particular,  single 
consequence  actions  forecasted  by  conventional 
models  do  not  work.  More  importantly,  con¬ 
ventional  models  yield  a  single  output  for 
multiple  inputs,  which  do  not  enable  predicting 
higher-level  and  derivative  consequences  for 
that  action.  In  order  to  improve  the  decision 
risk  process  it  is  suggested  that  higher-order 
consequence  models  au  proposed  by  J.  Coates 

[4]  be  adapted  for  integrated  split-brain 
holistic  decisionmaking  and  forecasting,  ffere 
research  and  development  into  such  models  would 
greatly  improve  our  decisions  to  forecasting 
for  an  uncertain  future. 

This  report  attempts  to  make  the  intelligent 
use  of  the  conscious  mind  more  palatable,  and 
of  the  unconscious  mind  more  respectable. 

David  Loye  states:  "...  Separation  of  the 
function  and  status  of  the  incredible  single 
power  of  the  human  mind  is  a  disturbing 
reflection  on  the  disjointedness  of  our  lives, 
thoughts,  and  times." 
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APPENDIX  A 


I 

HOW  CREATIVE  ARE  HOW? 


Researchers  have  developed  a  number  of  tests 
to  determine  whether  someone  is  predisposed  to 
think  creatively.  The  following  test  was 
designed  by  Eugene  Raudsepp  of  Princeton 
Creative  Research,  Inc.  after  studying  charac¬ 
teristics  of  highly  creative  people.  To  take 
the  test ,' indicate  ax  each  statement 
whether  you:  (A)  agree,  (B)  are  in-between  or 
don't  know,  or  (C)  disagree.  Answer  as  accu- 
rataly  and  frankly  as  possible,  trying  not  to 
guess  how  a  creative  person  might  respond. 
There  is  no  time  limit. 

1.  I  feel  that  a  logical  step-by-step 
method  is  best  for  solving  problems. 

2.  It  would  be  a  waste  of  time  for  me  to  ask 
questions  if  I  had  no  hope  of  obtaining 
answers. 

3.  I  always  work  with  a  great  deal  of  cer¬ 
tainty  that  I'm  following  the  correct 
procedures  for  solving  a  particular 
problem. 

4.  I  concentrate  harder  on  whatever  inter¬ 
ests  me  than  do  most  people. 

3.  When  trying  to  solve  a  proolem,  I  spend 
a  lot  of  time  analyzing  it. 

6.  I  occasionally  voice  opinions  in  groups 
that  seem  to  turn  some  people  off. 

7.  I  spend  a  great  deal  of  time  thinking 
about  what  others  think  of  me. 

8.  Complex  problems  and  situations  appeal  to 
me  because  I  find  them  challenging. 

9.  It  is  more  important  for  me  to  do  what  I 
believe  to  be  right  than  to  try  to  win 
the  approval  of  others. 

10.  People  who  seem  unsure  and  uncertain 
about  things  lose  my  respect. 

11.  More  than  other  people,  I  need  to  have 
things  interesting  and  exciting. 

12.  On  occasion  I  get  overly  enthusiastic 
over  things. 

13.  I  often  get  my  best  ideas  when  doing 
nothing  in  particular. 

14.  I  rely  on  intuitive  hunches  and  the  feel¬ 
ing  of  "rightness"  or  "wrongness"  when 
moving«toward  the  solution  of  a  problem. 


15.  I  sometimes  get  a  kick  out  of  breaking 
the  rules  and  doing  things  I'm  not 
supposed  to  do . 

16.  I  like  hobbies  that  involve  collecting 
things. 

17.  I  feel  I  have  capacities  that  have  not 
been  tapped  as  yet. 

18.  Daydreaming  has  provided  the  impetus  for 
many  important  projects. 

19.  I  like  poeple  who  are  objective  and 
rational. 

20.  I  see  myself  as  more  enthusiastic  and 
energetic  than  most  people  I  know. 

21.  I  can  get  along  more  easily  with  people 
if  they  belong  to  about  the  same  social 
and  business  class  as  myself. 

22.  I  have  a  high  degree  of  aesthetic  sensi¬ 
tivity. 

23.  I  have  a  highly  developed  capacity  for 
self-instruction. 

24.  I  like  people  who  are  most  sure  of  their 
conclusions . 

25.  Inspiration  has  nothing  to  do  with  the 
successful  solution  of  problems. 

26.  When  I'm  engaged  in  an  argument,  the 
greatest  pleasure  for  me  would  be  for  the 
person  who  disagrees  with  me  to  become  a 
friend,  even  at  the  price  of  sacrificing 
my  point  of  view. 

27.  I  tend  to  avoid  situations  in  which  I 
might  feel  inferior. 

28.  In  evaluating  information,  the  source  of 
it  is  more  important  to  me  than  the 
content . 

29.  I  resent  things  being  uncertain  and 
unpredictable. 

30.  I  like  people  who  follow  the  rule 
"business  before  pleasure." 

31.  One's  own  self-respect  is  more  important 
than  the  respect  of  others. 

32.  I  feel  that  people  who  strive  for  perfec¬ 
tion  are  unwise. 

33.  I  prefer  to  work  with  others  in  a  team 
effort  rather  than  solo. 

34.  I  believe  that  creativity  is  restricted  t 
specialized  fields  of  endeavour. 
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35.  It  is  important  for  me  to  have  a  place  55.  I  feel  it  is  important  to  understand  the 

for  everything  and  everything  in  its  motives  of  people  with  whom  I  have  to 

place.  deal. 

36.  Sometimes  I'm  sure  that  other  people  can  56.  I  can  see  things  in  terms  of  their 

read  my  thoughts.  potential. 


37.  The  trouble  with  many  people  is  that  they 
take  things  too  seriously. 

38.  I  have  a  great  deal  of  initiative  and 
self-starting  ability. 

39.  I  have  retained  my  serfse  of  wonder  and 
spirit  of  play. 

40.  I  can  maintain  my  motivation  and  enthusi¬ 
asm  for  my  projects,  even  in  the  face  of 
discouragement,  obstacles  or  opposition. 

41.  People  who  are  willing  to  entertain 
"crackpot"  ideas  are  impractical. 

42.  I'm  more  interested  in  what  could  be 
rather  than  what  is. 

43.  Even  after  I've  made  up  my  mind,  I  often 
can  change  it. 


57.  When  brainstorming  in  a  group,  I  am  able 
to  think  up  more  ideas  more  rapidly  than 
can  most  others  in  the  group. 

58.  I  am  not  ashamed  to  express  "feminine" 
interest  (if  man) ,  or  "masculine"  inter¬ 
ests  (if  woman)  if  so  inclined. 

59.  I  tend  to  rely  more  on  my  first  impres¬ 
sions  and  feelings  when  making  judgements 
than  on  a  careful  analysis  of  the  situa¬ 
tion  . 

60.  I  can  frequently  anticipate  the  solution 
to  my  problems. 

61.  I  often  laugh  at  myself  for  my  quirks  and 
peculiarities . 

62.  Only  fuzzy  thinkers  resort  to  metaphors 
and  analogies. 


44.  I  enjoy  fooling  around  with  new  ideas, 
even  if  there  is  no  practical  payoff. 

45.  I  think  the  statement,  "Ideas  are  a  dime 
a  dozen,"  hits  the  nail  on  the  head. 

”r  *  46.  I  don't  like  to  ask  questions  that  show 

ignorance. 


47.  Once  I  undertake  a  project,  I'm  deter¬ 
mined  to  finish  it,  even  under  conditions 
of  frustration. 


48.  I  sometimes  feel  that  ideas  come  to  me  as 
if  some  external  source  and  chat  I  am  not 
directly  responsible  for  them. 


49.  There  have  been  times  when  I  experienced 
an  "avalanche"  of  ideas. 


50.  I  try  to  look  for  ways  of  converting 
necessities  to  advantages. 

51.  It  is  wise  not  to  expect  too  much  of 
others. 


52.  I  am  able  to  more  easily  change  my  inter¬ 
ests  to  pursue  a  job  or  career  that  I  can 
change  a  job  to  pursue  my  interests. 

53.  Many  creative  breakthroughs  are  the 
result  of  chance  factors. 


63.  When  someone  triec  to  get  ahead  of  me  in 
a  line  of  people.  I  usually  point  it  out 
to  him. 

64.  Problems  that  do  not  have  clear-cut  and 
unambiguous  answers  have  very  little 
interest  for  me. 

65.  I  usually  work  things  out  for  myself 
rather  than  get  someone  to  show  me. 

66.  I  trust  my  feelings  to  guide  me  through 
experiences . 

67.  I  frequently  begin  work  on  a  problem  that 
I  can  only  dimly  sense  and  not  yet 
express . 

68.  I  frequently  tend  to  forget  things  such 
as  names  of  people,  streets,  highways, 
small  towns,  etc. 

69.  I  have  more  capacity  to  tolerate  frustra¬ 
tion  than  does  the  average  person. 

70.  During  my  adolescence  I  frequently  had  a 
desire  to  be  alone  and  to  pursue  my  own 
interests  and  thoughts. 

71.  I  feel  that  the  adage  "Do  unco  others..." 
is  more  important  chan  "To  thine  own  self 
be  true." 


54.  People  who  are  theoretically  oriented 
are  less  important  than  are  those  who 
are  practical. 


72.  Things  chat  are  obvious  to  others  are  not 
so  obvious  to  me. 
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73. 

I  feel  that  I  may  have  a  special  contri- 

1. 

-1,  0,  2 

38. 

2,  0,  -1 

bution  to  give  to  the 

world. 

2. 

0,  1,  2 

39. 

2,  0,  -1 

74. 

I  find,  that  I  have  more  problems  than  I 

3. 

0,  1,  2 

40. 

2,  0,  -1 

can  cackle,  more  work 

than  there  is  time 

4. 

3,  0,  -1 

41. 

-1,  0,  1 

for. 

5. 

2,  1,  0 

42. 

2,  1,  0 

75. 

Below  is  a  list  of  adjectives  and  terms 

6. 

2,  1,  C 

43. 

2,  1,  0 

that  aescrioe  people. 

indicate  wiin  a 

check  mark  ten  (10)  words  that  best 

7. 

-1,  0,  2 

44. 

2,  1,  0 

characterize  you. 

8. 

2,  1,  0 

45. 

-2,  0,  1 

energetic 

factual 

9. 

2,  0,  -1 

46. 

-1,  0,  1 

'  persuasive 

open-minded 

10. 

-1,  0,  2 

47. 

2,  0,  -1 

observant 

tactful 

fashionable 

inhibited 

11. 

2,  1,  0 

48. 

2,  0,  -1 

self-confident 

enthusiastic 

12. 

3,  0,  -1 

49. 

2,  1,  0 

persevering 

innovative 

forward-looking 

poised 

13. 

2,  0,  -1 

50. 

2,  0,  -1 

cautious 

acquisitive 

14. 

3,  1,  0 

51. 

1,  0,  -1 

habit-bound 

practical 

resourceful 

alert 

15. 

2,  1,  0 

52. 

-2,  1,  0 

egotistical 

curio us 

16. 

-1,  0,  1 

53. 

2,  1,  0 

independent 

organized 

good-natured 

unemotional 

17. 

2,  1,  0 

54. 

-2,  1,  0 

predictable 

clear-thinking 

18. 

3,  0,  -1 

55. 

2,  0.  -1 

formal 

understanding 

informal 

dynamic 

19. 

-1,  0,  1 

56. 

2,  0,  -1 

dedicated 

self-demanding 

20. 

2,  1,  0 

57. 

2,  0,  -1 

original 

polished 

quick 

realistic 

21. 

-1,  0,  1 

58. 

2,  1,  0 

efficient 

modest 

22. 

3,  1,  0 

59. 

1,  0,  -I 

helpful 

involved 

perceptive 

absent-minded 

23. 

2,  1,  0 

60. 

2,  1,  0 

courageous 

flexible 

24. 

-1,  0,  1 

61. 

2,  0,  -1 

stem 

sociable 

tho rough 

well-liked 

25. 

-2,  0,  2 

62. 

-2,  0,  2 

impulsive 

restless 

26. 

-1,  0,  1 

63. 

2,  1,  0 

determined 

retiring 

27. 

-1,  0,  1 

64. 

-1,  0,  1 

28. 

-2,  1,  2 

65. 

1,  0,  -1 

ANSWERS 

TO 

29. 

-1,  0,  1 

66. 

2,  1,  0 

CREATIVITY 

TEST 

30. 

-1,  0,  1 

67. 

2,  1,  0 

31. 

2,  1,  0 

68. 

2,  0,  -1 

Scoring  instructions:  To 

compute  your  score, 

32. 

-2,  0,  1 

69. 

2,  1,  0 

add 

up  the  points  assigned 

to  each  item.  For 

33. 

-1,  1,  2 

70. 

2,  0,  -1 

each 

i  question,  the  first  value  is  for  A  (agree) 

the 

second  is  for  B  (in-between  or  don' t  know) 

34. 

-1,  0,  1 

71. 

-1,  0,  1 

and 

the  third  Is  for  C  (disagree). 

35. 

-1,  0,  1 

72. 

2,  1,  0 

36. 

-2,  0,  2 

73. 

1,  0,  -1 

37. 

-1,  0,  1 

74. 

2.  1,  0 
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75.  The  following  have  values  of  2: 

energetic,  observant,  persevering, 
resourceful,  independent,  dedicated, 
original,  perceptive,  courageous, 
enthusiastic,  innovative,  currious, 
dynamic,  self-demanding,  involved, 
flexible . 

The  following  have  values  of  1:  self- 
confident,  forward-looking,  informal, 
thorough,  open-minded,  alert,  restless, 
;Ji“" J  determined. 

i 

The  rest  have  values  of  0. 


125-150 

Exceptionally  Creative 

90-124 

Very  Creative 

55-89 

Above  Average 

35-54 

Average 

15-34 

Below  Average 

-56-14 

Noncreative 
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